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Executive  Summary 


The  Defense  Acquisition  System,  as  documented  in  DoDI  5000.02,  mandates  that  programs 
should  not  follow  process  models  by  rote.  Rather,  stakeholders  should  tailor  program  activities 
and  documentation  according  to  specific  requirements,  priorities,  risks,  and  boundary 
conditions.  Adaptive  acquisition  provides  an  evolving  set  of  broadly  applicable  adaptive 
engineering  and  adaptive  procurement  tools  and  processes  intended  to  help  tailor  acquisition 
programs  per  the  following  approach: 

•  Specify  all  critical,  lifecycle,  mission,  system,  and  process  objectives,  i.e.  “360° 
requirements" ,  as  measures  of  performance/effectiveness  within  executable  test  cases. 

•  Obtain  commitments  by  all  stakeholders  to  the  tailored  approach  upfront. 

•  Apply  robust  360°  Validation  and  Verification  (V&V)  to  select  best  existing  capability  (i.e. 
use  mature,  preferably  COTS,  technology  for  prototyping,)  benchmark  the  state  of  the 
art  as  an  X%  solution,  and  measure  progress  going  forward. 

•  Design  systems  and  plan  system  engineering  to  modularize,  connect,  and  deploy  best 
available  existing  capability.  Develop  and  deploy  incremental  improvements  in  a 
virtuous  lifecycle  process.  Apply  Modular  Open  System  Approaches  (MOSA)  as 
appropriate. 

•  Tailor  procurement  vehicles  to  lower  barriers  of  entry;  incentivize  broad  competition 
and  teaming;  reduce  solicitation  to  award  timelines. 

•  In  cases  where  contracting  under  the  Federal  Acquisition  Regulations  is  too  restrictive, 
apply  "Other  Transactions"  Authority  (10USC2371b)  to:  establish  open  consortium  of 
pre-vetted  traditional  and  non-traditional  competitors  and  collaborators;  streamline 
cost  accounting  and  competitive  process;  align  intellectual  property  rights;  make  direct 
award  to  transition  developed  capability  to  production. 

•  Streamline  statutory  and  regulatory  bureaucratic  compliance  documentation  to 
concisely  capture  rationale  for,  plans  for,  and  achievement  of,  the  above. 
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Motivation  and  Background 

The  Secretary  of  the  Air  Force  established  the  Bending  the  Cost  Curve  (BTCC)  initiative  to  identify, 
nurture,  and  broadly  instantiate  processes  and  practices  to  make  weapon  systems  acquisition  more 
efficient  and  effective.  The  Assistant  Secretary  of  the  Air  Force  for  Acquisition  (SAF/AQ)  established  its 
Office  of  Transformational  Innovation  (OTI)  to  (among  other  things)  manage  BTCC.  OTI's  research 
confirms  that  program  offices,  in  general,  are  reluctant  to  depart  from  one-size-fits-all  approaches  to 
compliance  with  acquisition  policy.  That  reluctance  is  in  spite  of  Federal  Acquisition  Regulations  (FAR), 
Defense  FAR  (DFAR),  DoDI  5000.02,  and  recent  Better  Buying  Power,  Performance  Based  Logistics  (PBL), 
and  "Should  Cost/Will  Cost"  policy  guidance.  This  guidance  clearly  explains  that  programs  should  tailor 
the  standard  acquisition  models  to  align  with  the  details  of  their  requirements  in  order  to  maximize 
value  returned  per  unit  of  time  and  money  invested.  Likewise,  acquisition  policy  suggests  applying 
Modular  Open  System  Approaches  (MOSA)  to  enable  business  models  that  leverage  the  inherent 
adaptability  of  plug-and-play  designs.  Recent  Congressional  language  reiterates  and  in  some  cases 
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expands  authority  to  innovate  within  the  acquisition  process.  In  particular,  recent  National  Defense 
Authorization  Act  (NDAA)  language  emphasizes  use  of  Rapid  Prototyping,  Rapid  Fielding,  and  "Other 
Transactions"  Authority  (OTA.)  However,  this  policy  stops  short  of  providing  detailed  implementation 
guidance.  Accordingly,  OTI  is  compiling  a  continuously  evolving  adaptive  acquisition  framework  to 
provide  an  evolving  set  of  broadly  applicable  tools  and  processes  intended  to  help  program  managers, 
and/or  prime  contractors,  to  tailor  their  programs. 

Adaptive  Acquisition  in  a  Nutshell 
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Figure  1:  Virtuous  cycle  of  incremental  continuous  improvement  +full  horizon  of  requirements,  boundary  conditions,  and 
solutions  =  360°  adaptive  perspective  on  acquisition 


The  360°  View  of  Adaptability 

In  general,  "adaptability"  is  the  ability  to  effectively  react  to  circumstances.  In  the  context  of 
acquisition,  "adaptability"  means  to  reduce  risk  to  cost,  performance,  and  schedule,  by  reacting  to 
circumstances.  In  a  sense,  adaptability  is  the  opposite  of  rigidity.  Rigidity,  in  context  with  acquisition 
process,  is  associated  with  prescribed,  long,  linear,  piecemeal,  processes;  with  hierarchal  management 
structure;  that  emphasize  bureaucratic  compliance  with  policy.  It  follows  that  adaptive  acquisition 
would  be  characterized  with  situationally-dependent,  relatively  short  iterative  cycles,  addressing 
multiple  opportunities  and  concerns  in  parallel;  with  relatively  flat  management  structure;  that 
emphasize  achieving  desired  outcomes  with  minimally  essential  documentation.  Thus  an  adaptive 
approach  to  acquisition  should  embrace  the  concepts  of:  1)  a  virtuous  360°  cycle  of  iterative, 
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continuous,  improvement;  and  2)  simultaneous  consideration  of  a  360°  view  of  many  desired  outcomes, 
stakeholders,  and  potential  solutions.  (See  figure:  1.) 

"Adaptive  acquisition"  therefore  includes  exposing  outcome-based  requirements  to  as  broad  a 
marketplace  of  solution  providers  as  possible;  benchmarking  of  best  existing  capability;  reactively 
adapting  system  design  to  take  advantage  of  existing  mature  technology;  and  streamlining  engineering, 
programmatic,  and  procurement  bureaucracy  accordingly.  Hence,  adaptive  acquisition  begins  with 
specification  of  measurable  and  testable  objectives  for  all  aspects  of  the  targeted  capability.  Relevant 
aspects  for  improvement  include:  system  performance;  lifecycle  cost,  tech  refresh  cycle,  reliability,  and 
maintainability;  training;  certifiability  for  safety  and  environmental  factors;  acquisition  process  efficiency 
and  effectiveness  including  speed-to-capability,  and  capability-per-cost.  Given  clearly  specified 
measures  of  performance  and/or  effectiveness  for  all  these  parameters  -  i.e.  a  360°  Statement  of 
Objectives  and  measures  (360°  SOO)  —  adaptive  acquisition  applies  two  perspectives  to  address  them: 
adaptive  engineering,  and  adaptive  procurement. 

Adaptive  Engineering 

Adaptive  engineering  embraces  modularity  and  openness  in  the  sense  that  it  recognizes  that  building 
systems  by  integrating  existing,  mature,  and  trusted  components  or  subsystems  is  one  very  effective 
approach  to  managing  risk  across  system  lifecycles.  Adaptive  acquisition  also  recognizes  that  robust 
test-based  Validation  and  Verification  (V&V)  is  essential  to  managing  risk  regardless  of  the  risk  profile. 
"Validation"  means  confirmation  that  achieving  threshold  Measures  of  Performance  (MoP)  will  lead  to 
achieving  threshold  Measures  of  Effectiveness  (MoE)  for  the  360°  targeted  program  outcomes. 
"Verification"  means  confirmation  that  specified  MoP  are  achieved. 

Thus,  having  identified  360°  objective  threshold  measures  of  performance  and  effectiveness,  adaptive 
engineering  requires  specifying  the  test  cases  for  all  system-related  parameters.  The  set  of  test  cases  are 
used  to  specify  a  conceptual  test  bench  that  will  provision  an  instance  of  the  end-to-end  target 
architecture  together  with  test  tools  aligned  with  all  objectives.  Developing  a  prototype  test  bench  is 
often  the  first  engineering  task.  The  program  office  should  provision  the  test  bench  itself,  or  at 
minimum  it's  specifications,  as  Government  Furnished  Equipment  (GFE)  to  all  potential  bidders.  Bids 
then,  take  the  form  of  proposed  prototyping  projects  that  address  the  government's  Statement  of 
Objectives  for  component,  subsystem,  or  system  technology  and/or  processes. 

In  this  sense  a  "prototype"  is  simply  a  design  model.  Prototypes  can  be  physical  or  virtual  and  represent 
technology  or  processes.  Prototypes  can  be  mature  or  developmental.  Generally,  adaptive  acquisition 
aims  to  identify  and  perform  baseline  validation  and  verification  of  candidate  prototypes.  The  more 
mature  the  prototype  the  better  -  lifecycle  supported  COTS  is  the  ideal  case.  Regardless,  following  initial 
V&V,  prototype  development  aims  to  close  the  gap  between  existing  capability  and  all  threshold 
requirements.  This  360°  V&V  (i.e.  objective  evaluation  of  process  and  system  performance  in  context 
with  all  lifecycle  objectives  for  program  outcomes)  provides  basis  for  tailoring  Requirements  Reviews 
(RR)  and  Design  Reviews  (RR).  Indeed,  an  RR  or  DR  at  any  scope  or  level  of  maturity  takes  the  form  of 
analysis  of  the  results  of  associated  objective  V&V  and  need  not  include  exhaustive  and  subjective  pro 
forma  review  of  boilerplate  topics.  Exit  criteria  for  RR  or  DR  is  simply  tested  achievement  of  threshold 
criteria  across  the  360o  view  of  process  and  system  MoP  and  MoE. 
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Adaptive  Procurement 


Adaptive  procurement  starts  by  specifying  requirements  for  procurement  process  efficiency  and 
effectiveness  in  measurable  and  testable  terms  such  as  time-to-award,  lifecycle-cost-per-capability; 
numbers  and  quality  of  traditional  and  non-traditional  competitors;  maturity  and  robustness  of  offered 
solutions;  etc.  Adaptive  acquisition  then  exercises  the  letter  and  spirit  of  regulatory  and  statutory  policy 
to  tailor  parameters  such  as  cost  accounting,  intellectual  property  (IP)  agreements,  scope  of 
competition,  basis  of  source  selection,  compensation  models,  and  associated  bureaucratic  compliance 
as  appropriate. 

In  cases  where  traditional  FAR-based  contracting  is  not  sufficiently  flexible  to  achieve  tailoring 
objectives,  adaptive  procurement  e mploys  new  statutory  authorities  regarding  rapid  prototyping,  rapid 
fielding,  and  use  of  "Other  Transactions"  for  Prototyping  Projects.  Use  of  "Other  Transactions" 

Authority  (OTA)  for  Prototyping  Projects,  which  is  not  subject  to  the  Federal  Acquisition  Regulations 
(FAR),  is  particularly  powerful.  For  example  the  program  office  can  competitively  awarded  an 
"umbrella"  OT  to  an  open  consortium  of  traditional  and  non-traditional  Defense  contractors  with 
specified  funding  ceiling  and  period  of  performance.  The  consortium  manager  accepts  risk  for  vetting 
members.  Thus,  consortium  managers  should  be  fire-walled  from  participating  in  the  actual  funded 
project  work,  but  they  may  be  for-profit  or  not-for-profit  firms  or  individuals.  Virtually  any  firm  willing 
to  sign  a  simple  charter  agreement  may  join  quickly  and  easily.  Membership  in  the  consortium  makes 
the  firm  a  qualified  Defense  contractor.  The  program  office  can  exercise  whatever  reasonable  approach 
to  cost  accounting,  IP,  incentive  model  etc.  it  feels  is  appropriate  to  very  quickly  (weeks  not  months) 
award  funds  to  develop  and/or  demonstrate,  and  V&V  relevant  prototypes  of  components,  subsystems, 
systems,  and  financial  models.  When  prototypes  achieve  exit  criteria,  the  program  office  may 
immediately  award  a  traditional  contract  or  OT  for  production. 


Steps  of  the  Adaptive  Acquisition  Process 


1.  Establish  360°  partnerships.  Identify  the  broad  stakeholder  community,  i.e.  operator,  procurement 
authorities,  industrial  organizations,  test  and  evaluation  authorities,  certification  authorities,  and  legal 
authorities  critical  to  end-to-end  program  success.  Identify  resources  required  within  each  stakeholder 
community.  Establish  agreed  "virtuous  cycle"  feedback  loop  necessary  to  achieve  process  outcomes 
described  below. 

2.  Prepare  a  360°  Statement  of  Objectives  (SOO).  Specify  the  parameters  of  all  important  project 
lifecycle  outcomes  -  operational  performance,  technical  performance,  interoperability,  security,  safety, 
technical  refresh,  lifecycle  cost,  training,  certifiability,  etc.  --  in  measurable  ways. 

3.  Specify  360°  Measures  of  Performance/Effectiveness  (MoP/E).  Specify  test  cases  that  deliver  MoP 
and/or  MoE,  including  acceptable  threshold  values,  for  each  of  the  important  parameters. 

4.  Employ  360°  procurement  vehicles.  Plan  and  prepare  procurement  vehicles  that:  both  encourage  and 
streamline  broad  competition  and  teaming;  employ  360°  SOO  as  basis  of  selection  and  incentives; 
streamline  cost  accounting,  and  focus  intellectual  property  agreements  according  to  project  priorities; 
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and  allow  parallel  and  symbiotic  execution  of  developmental,  procurement,  and  sustainment  funds.  If 
Federal  Acquisition  Regulations  (FAR)  preclude  any  of  these  objectives,  employ  "Other  Transactions" 
Authority  to  achieve  greater  flexibility. 

5.  Provision  a  360°  test  bench.  Design  and  develop  test  resources  that  address  all  objectives,  and  make 
them  as  automated,  integrated,  and  broadly  available  -  perhaps  virtually  via  cloud  technology  —  to  as 
many  potential  solution  providers  as  possible.  Test  bench  must  provision  all  existing  architectural 
boundary  conditions  and  any  other  government  furnished  resources.  Test  bench/process  should  deliver 
artifacts  required  for  certification  to  the  extent  practicable. 

6.  Benchmark  the  360° COTS  baseline.  Use  all  available  crowd  sourcing  channels  to  expose  the  360°SOO 
and  solicit  the  broadest  possible  community  COTS  providers  to  demonstrate  existing  products  and 
services  as  prototype  solutions  for  all  or  some  of  project  objectives.  Perform  360°  testing  to  down  select 
best-of-breed.  Perform  Validation  and  Verification  (V&V)  to  benchmark  -  and  as  much  as  possible, 
certify  —  the  specific  ability  of  existing  off-the-shelf  technology  to  satisfy  360°  objectives. 

7.  Align  360°  solution  architecture  with  benchmarked  reality.  Use  knowledge  of  performance  of  existing 
products  and  services  to  compose  an  optimum  solution  architecture,  or  modify  an  existing  solution 
architecture,  by  "connecting"  best-of-breed  capabilities.  Tenets  of  MOSA  apply. 

8.  Tailor  regulatory  and  statutory  governance  from  a  360°  adaptive  perspective.  Designate  the  360°SOO 
as  the  "Tailored  Requirements  Document."  If  it  is  possible  to  satisfy  360°  requirements  through 
adaptive  lifecycle  tech  refresh  of  an  existing  program,  do  so.  Otherwise,  document  the  adaptive 
acquisition  strategy,  AoA,  cost  analysis,  logistic  support  planning  performed  in  steps  above  as  exit/entry 
criteria  for  all  or  part  of  the  required  Material  Solution  Analysis  (MSA)  and  Technology  Maturation  and 
Risk  Reduction  (TMRR).  Document  the  360°  virtuous  engineering  cycle  of  incremental  mature 
technology  prototype  V&V  =>  deploy  =>  incremental  improvement  =>  V&V  =>  deploy  as  an  adaptive 
Systems  Engineering  Plan  (SEP)  and  Test  &  Evaluation  Master  Plan  (TEMP)  for  Adaptive  Engineering  and 
Material  Development  (EMD).  Document  plans  for  use  of  360°  procurement  vehicles,  whether  based  on 
FAR  or  OTA,  as  the  Acquisition  Plan.  Brief  Acquisition  Strategy  Plan  (ASP)  accordingly,  and  obtain 
Decisions  &  Findings  (D&F)  (e.g.  for  use  of  OTA)  and/or  waivers  as  necessary.  (See  figure:  2.) 

9.  Field  X%  of  the  ideal  360°  solution  immediately.  Compose  a  certified  implementation  of  the  solution 
architecture  from  mature  components.  Execute  procurement  or  sustainment  funds  necessary  to  field 
the  X%  solution. 

10.  Perform  360°  virtuous  cycle  prototype  development  to  close  gap  across  the  lifecycle.  Precisely 
define  the  requirements  gap  between  benchmarked  existing  capability  and  the  next  incremental 
objective.  Execute  RDT&E  funds  to  allow  COTS  solution  providers  to  improve  existing  mature 
prototypes  (i.e.  current  versions  of  COTS  offerings.)  Assure  that  intellectual  property  rights  are 
sufficient  to  achieve  government  objectives  regarding  use  within  the  program  element,  and  reuse  across 
program  elements,  across  the  lifecycles  of  interest. 

11.  Iterate  steps  1-10  across  360°  of  capability  lifecycle. 
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Adaptive  Acquisition  in  DODI  5000  Context 


These  are  all 
"prototypes."  The 
more  mature  the 
first  prototype  (e.g. 
COTS),  the  less 
time/effort  required 
to  get  to  production, 
and  less  risk  to 
lifecycle  costand 
performance. 


and  Risk  Reduction 


A 


)•  COD  Validation) 


^Development  RFP  Release 

Development  Contract  Award 
.(OoO:  If  " 

A= 


Proceeding  to  LRIP  without 
having  evaluated  a  robust  model, 
i.e.  a  mature  prototype,  is 
unnecessarily  risky  and 
expensive,  but  common  under 
traditional  approaches. 


Development 


_~1 - swwJA 


Adaptive  Acquisition  by  Prototype 

Requirements:  emphasize  outcomes,  not 
design,  and  trade  space  across  lifecycle 
TEMP:  Leverage  mature,  well-documented, 
prototypes  to  streamline  testing 
Risk  Matrix: 

•  Streamlined  process  reduces  risk  to 
schedule 

•  Mature  prototype  V&V  reduces  risk  to 
cost,  technical  performance,  and 
schedule 

•  OTA  reduces  risk  of  protest  &  risk  of 
vendor  lock 

SRR:  Preliminary  V&V  of  prototype  provides 

basis  of  source  selection  for  OTA 

prototyping  awards 

PDR:  Robust  V&V  down  selects  viable 

solutions 

CDR=  Integration  test  of  mature  prototype 
(perhaps  in  lieu  of  LRIP):  Exit  criteria  = 
production  decision  criteria  for  direct 
award  of  follow  on  production 


Figure  2:  Adaptive  acquisition  is  way  to  approach  tailoring  DoD  5000.02 

Adaptive  Engineering  Procedures  to  Optimize  Risk-Reward 

The  following  steps  aim  to  add  implementation  detail  to  the  engineering  aspects  of  the  conceptual 
description  of  Adaptive  Acquisition  provided  above. 


1.  Consider  if  and  how  to  apply  Modular  Open  System  Approaches  (MOSA)  to  enhance  programmatic 
lifecycle  efficiency  and  effectiveness.  (See  Open  System  Practical  Guide) 

2.  Use  of  MOSA  notwithstanding,  parse  program  into  a  portfolio  of  quasi-independent,  relatively  short 
duration,  engineering  tasks  -  typically  aimed  at  evolving  prototype  technology  and/or  processes  -- 
and  periodic  integration  events,  with  clear,  objective,  exit  criteria.  (See  figure:  3.) 

3.  Specify  testable,  outcome-based  threshold  and  objective  requirements.  Identify  associated  risk- 
reward  factors  and  management  actions.  (See  Appendix  A.) 

4.  Adjust  these  requirements  and  risk-reward  management  plan  iteratively  and  frequently.  Address, 
at  minimum,  the  following  topics. 

4.1.  Mission  effectiveness 

4.2.  System  performance 

4.3.  Lifecycle  speed-to-capability  (initial  and  tech  refresh) 

4.3.1.Time  to  procurement  award 

4. 3. 2. Incremental  development  cycle  time 

4.3.3. Time  to  test 

4.3.4. Time  to  certify 

4.4.  Lifecycle  cost-per-capability 

4.5.  Safety/flight  worthiness,  including  certifiability 
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4.6.  Enterprise  interoperability 

4.6.1.  Cyber  security  and  certifiability 

4. 6. 2.  Position  Navigation  and  Time  (PNT) 

4. 6. 3.  Component  reusability 
4. 6. 4. Information  sharing 
4.6.5.Training 

5.  Create  360°  test  cases  and  weighting  factors  that  address  all  the  objectives  specified  per  the  above. 
Specify  these  test  cases  and  weighting  factors  as  basis  of  "Value  Adjusted  Total  Evaluated  Price" 
(VATEP)  source  selection  criteria. 

6.  Provision  any  Government  Furnished  Information/Equipment  (GFI/E)  that  might  serve  as  physical  or 
virtual  models,  i.e.  "prototypes",  of  technology  or  process  objectives. 

7.  Publish  360°  test  cases,  together  with  summary  of  overall  acquisition  project  parameters  such  as 
budget,  schedule,  competitive  considerations,  risk-reward  management  strategy,  access  to  GFI/E, 
and  any  architectural  constraints  as  broadly  as  possible.  Do  not  specify  particular  engineering 
solutions. 

8.  Provision  360°  test  cases.  Solicit  all  potential  solution  providers  to  Validate  and  Verify  (V&V)  their 
offered,  mature,  physical  or  virtual  models  that  address  all  mission,  technological,  and  process 
objectives. 

9.  Apply  V&V  results  to 

9.1.  Perform  trades  analysis  and  adjust  requirements. 

9.2.  Specify  the  requirements  gap  between  tested  best  of  breed  off-the-shelf  capability  and 
adjusted  threshold  and  objective  requirements. 

9. 2.1. Use  this  gap  analysis  to  specify  developmental  exit  criteria  (e.g.  M/S  C)  prior  to  award  for 
production. 

10.  Publish  V&V  outcomes  as  broadly  as  possible.  Award  funds  to  best-of-breed  solution  provider(s)  to: 

10.1.  If  developmental  exit  criteria  are  not  met,  execute  RDT&E  as  necessary  to  close  a 
specified  increment  of  capability  requirement  gap,  or 

10.2.  If  developmental  exit  criteria  are  met,  provision  specified  quantities  of  lifecycle- 
supported  capability. 
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Structure  of  Prototype  Project 


Statement  of  Objectives  (SOO)  =>  Request  for  Prototype  (RFP) 


Requirement  Statement 

, _  i 

r 

•  Lifecycle  military  performance 

•  Lifecycle  tech  performance 

•  Lifecycle  schedule  performance  * 

•  Lifecycle  cost  performance 

•  Lifecycle  maintenance 
performance 

•  Lifecycle  training  performance 

•  Lifecycle  cyber  security 
performance 

•  Lifecycle  flight  worthiness 
performance 

•  Intellectual  property 
performance 


Metrics 

Thresholds 

Objectives 

P3I 

Trade  space 


Prototyping  Activity 


Prototyping  Outcome 


I 

•  Use  Case  •  Validate  • 

•  Mission  threads 

•  Prototype  In  *  Verify  • 

lieu  of 

•  SRR?  • 

•  PDR? 

•  CDR? 

•  LRIP? 


Test 

Evaluation 

Certification 
of  design 


Decision  to 
manufacture 

Decision  to  do 
more 

Prototyping 


•  Test  Case 

•  Model 

•  Virtual 

•  Physical 

•  Process 

•  Capability 

•  Simulation 


Figure  3:  Each  prototyping  project  manager  should  consider,  at  minimum,  the  elements  portrayed  in  this  diagram. 

Adaptive  Procurement  Procedures  to  Align  Competition  and  Incentives 

Competitive  procedures,  as  mandated  by  the  FAR,  aim  to:  a)  attain  "best  value",  i.e.  optimize  capability- 
per-cost  for  the  government  through  competition  across  the  industrial  base;  and  b)  assure  that 
government  funds  are  allocated  equitably.  However,  effectiveness  of  competitive  procedures  under  the 
FAR  often  suffers  from  burdensome  serial,  repetitive,  and  subjective  processes  that  tend  to  take  a  long 
time,  and  preclude  participation  by  potential  solution  providers  who  are  not  willing  to  suffer  what  they 
perceive  as  undue  bureaucracy.  Adaptive  acquisition  aims  to  attain  best  value,  lower  barriers  of  entry, 
and  decrease  timelines  by: 

Reducing  redundant  paperwork; 

Parallelizing  processes; 

Increasing  transparency  of  budgets  and  schedules; 

Catalyzing  formation  of  open  consortia  of  competitors  and  collaborators; 

Employing  objective  test  cases  of  both  technical  performance,  and  lifecycle  acquisition 
processes,  as  basis  of  due  diligence,  selection,  and  awards. 

The  following  steps  aim  to  add  implementation  detail  to  the  procurement  aspects  of  the  conceptual 
description  of  Adaptive  Acquisition  provided  above. 

1.  Achieve  outreach,  transparency,  and  efficiency  through  open,  program-centric,  consortia  or  sub¬ 
consortia.  Expand  the  concept  of  "Industry  Day"  to  catalyze  a  persistent  community  of  traditional 
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and  non-traditional  contractors.  Preferably,  incorporate  as  a  not-for-profit  organization  with  low 
barriers  to  entry. 

1.1.  Leverage  existing  not-for-profit  organizations  as  appropriate.  For  example,  explore 
feasibility  of  establishing  a  working  group  within  existing  trade  associations  or  "Other 
Transactions"  Agreement  (OTA)  consortia. 

1.2.  Specify  that  the  purpose  of  program-centric  consortium  is  to  facilitate  government-industry 
collaborative  engineering,  and  equitably  facilitate  competition  and  government  investment 
throughout  the  lifecycle  of  the  program  or  project  of  interest. 

1.2.1.  Establish  frequently  iterative  feedback  process  to  discuss  requirements,  budgets, 
schedules,  and  potential  solutions;  issue  solicitations;  and  receive  suggestions. 

1.2. 1.1.  Consider  use  of  consortium-type  OTA,  or  executing  some  other  contractual 
relationship  with  the  consortium,  as  means  to  compete  and  award  parallel 
engineering  activities  across  program/project  lifecycle. 

1.3.  Employ  360°  test  cases  as  basis  of  competition,  initial  and  incentive  awards,  and  contract 
language  generally. 

1.3.1.  Plan  to  attain  Best  Value  through  Trade  Off  via  VATEP.  (In  some  cases  trade  off  analysis 
may  lead  to  conclusion  that  the  best  option  is  lowest  cost,  technically  qualified  option.) 

1.3.2.  Create  concise  360°  "Statement  of  Objectives"  (SOO)  comprised  of  the  requirements  and 
test  cases.  Adjust  the  SOO  continuously  at  each  iterative  programmatic  event. 

1.3. 3.  Conduct  market  survey  and  trades  analysis  by  publishing  the  SOO  broadly  and  is  far  in 
advance  of  intended  award  as  is  practical.  Invite  respondents  to  demonstrate  existing 
capability  per  test  case.  Adjust  SOO  per  lessons  learned. 

1.3.4.  Solicit  proposals  for  parallel  engineering  tasks. 

1. 3.4.1.  Specify  incentive-based  funding  ceiling  for  each  task,  e.g.  model  similar  to  "Cost 
Plus  Fixed  Fee/Award/Incentive"  or  grant.  (OTs  are  not  constrained  by  FAR  models, 
but  may  certainly  leverage  effective  approaches.) 

1.3. 4.1.1.  If  an  OTA  is  used  in  lieu  of  FAR  contract,  the  fee  structure  might  be 
tailored  beyond  typical  FAR  models.  E.g.,  the  "plus"  might  be  a  negative 
number  in  cases  where  vendors  agree  to  share  project  costs.  "Incentives" 
might  be  in  the  form  of  favorable  intellectual  property  rights  rather  than  a 
monetary  award. 

1.3. 4. 2.  Specify  that  V&V  of  existing  capability  will  serve  as  sole  basis  of  both  Best  Value 
(most  likely  VATEF)  source  selection  criteria,  and  negotiation  of  contractors'  bid. 

(E.g.,  the  contractors'  submitted  lifecycle  capability-per-cost  model  will  constitute 
the  contractors'  bid.  V&V  of  that  model  will  constitute  government  due  diligence.) 

1.3. 4. 2.1.  If  an  OTA  is  used  in  lieu  of  FAR  contract,  the  Truth  in  Negotiations  Act 
(TINA)  may  not  apply.  Nevertheless,  USG's  V&V  of  proposed  cost  model  must 
be  sufficiently  rigorous  to  stand  auditor's  scrutiny  in  context  with  size  of  award, 
and  risk  to  project  success. 
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1.3. 5. Specify  objective  V&V  criteria,  for  any  or  all  of  the  topics  under  adaptive  engineering 
step  4  that  qualify  for  initial  award  or  follow-on  incentives. 

1.3.6.Award  appropriately  incentivized  contract  or  OTA  for  each  parallel  engineering  task  solely 
on  basis  of  V&V  events. 

1.3. 6.1.  Use  SOO  as  basis  of  PWS. 

1.3. 6. 2.  Monitor  contractor  performance  as  part  and  parcel  of  periodic  V&V  events  and 
make  or  withhold  payment  as  appropriate. 


Further  Detail 


Appendices  A-C  provide  more  detail.  Appendix  A  provides  background,  step  by  step  procedures,  and 
templates  for  soliciting  and  awarding  "Other  Transactions."  Appendix  B  explains  how  to  measure  value 
in  context  with  Defense  systems,  and  therefore  optimize  risk  mitigation  activities  to  maximize  return  on 
investment.  Appendix  C  provides  selected  references  to  relevant  adaptive  acquisition  policy  together 
with  excerpts. 


Chris  Gunderson/SAF/AQ  OTI/(o)703-693-4177/(m)831-224-5182/25July2017 


14 


Appendix  A:  “Other  Transactions"  Practical  Guide 


"Other  Transactions" 
Practical  Guide 


This  a  "living"  informal  collection  of  lessons  and  learned  and  effective  practices.  It  is  not  an  official 
document,  and  does  not  represent  U.S.  Government  Policy. 


Assistant  Secretary  of  the  Air  Force  for  Acquisition 
Office  of  Transformational  Innovation 
9  January,  2017 
(Update  24  July,  2017) 
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Official  Guidance 

Please  see  the  Office  of  the  Secretary  of  Defense  (OSD)  "Other  T ransactions"  Guide  for  Prototype 
Projects  of  January  2017  (embedded  at  page  54)  for  official,  but  also  thorough  and  pragmatic,  guidance 
for  application  of  "other  transactions"  authority  within  the  Department  of  Defense.  This  unofficial  OT 
Practical  Guide  aims  to  be  entirely  consistent  with  the  official  guidance,  but  also  thoroughly  informed  by 
extensive  interaction  with  actual  practitioners. 

What  is  “Other  Transactions"  Authority  (OTA)? 

Under  US  Code  Title  10  (10  USC  2371  and  2371b),  an  "other  transaction,"  or  just  "transaction,"  is  a 
legally  binding  procurement  agreement  between  government  and  industry  that  is  not  governed  by  rules 
for  "contracts"  per  the  Federal  Acquisition  Regulation  (FAR).  Congress  established  "Other  Transactions" 
Authority  in  recognition  that,  in  order  to  achieve  the  US  Government's  (USG)  objectives  for  innovation, 
the  USG  must,  from  time  to  time,  depart  from  prescriptive  procurement  boilerplate,  which  today  is 
represented  by  the  FAR.  For  example  FAR  bureaucratic  requirements  regarding  Cost  Accounting 
Standards  (CAS),  Intellectual  Property  Rights  (IPR),  and  Competitive  Procedures  might  be  at  odds  with 
government  requirements  to  engage  and  incentivize  non-traditional  partners,  accelerate  speed-to- 
capability,  evaluate  alternative  business,  mission,  or  engineering  processes,  etc.  In  those  cases, 
government  officials  are  trusted  to  respect  the  intent  of  the  underlying  legislation  that  led  to  FAR 
guidance,  but  are  empowered  to  impose  alternative  methods  to  achieve  that  intent  that  are  appropriate 
to  the  specific  risk,  priority,  and  expertise  profile  of  the  project  of  interest.  Fourteen  federal  agencies 
have  OTA.  The  first  to  receive  it  was  NASA  in  1958.  Congress  granted  the  authority  for  early  phase 
research  to  DARPA  in  1989,  and  extended  it  to  all  of  DoD  and  expanded  the  scope  to  include  prototyping 
projects  in  1994.  Congress  has  steadily  expanded  the  scope  of  the  statue  over  the  years.  See  excerpts 
below: 


10  USC  2371  -  Research  projects:  transactions  other  than  contracts  and  grants 

"...  The  Secretary  of  Defense  and  the  Secretary  of  each  military  department  may  enter 
into  transactions  (other  than  contracts,  cooperative  agreements,  and  grants) ...  A 
cooperative  agreement  containing  a  clause  under  subsection  (d)  or  a  transaction 
authorized  by  subsection  (a)  may  be  used  for  a  research  project  when  the  use  of  a 
standard  contract,  grant,  or  cooperative  agreement  for  such  project  is  not  feasible  or 
appropriate..." 

10  USC  2371b  -  Authority  of  the  Department  of  Defense  to  Carry  out  Certain  Prototype 
Projects 
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"...  Secretary  of  Defense  may, ...  carry  out  prototype  projects  that  are  directly  relevant 
to  enhancing  the  mission  effectiveness  of  military  personnel  and  the  supporting 
platforms,  systems,  components,  or  materials  proposed  to  be  acquired  or  developed  by 
the  Department  of  Defense,  or  to  improvement  of  platforms,  systems,  components,  or 
materials  in  use  by  the  armed  forces....  for  a  prototype  project ...  not  in  excess  of 
$250,000,000  upon  a  written  determination  by  the  senior  procurement  executive  ... 

(that)  use  of  the  authority  of  this  section  is  essential  to  promoting  the  success  of  the 
prototype  project...  To  the  maximum  extent  practicable,  competitive  procedures  shall 
be  used  when  entering  into  agreements  to  carry  out  projects  ...  prototype  project  may 
provide  for  the  award  of  a  follow-on  production  contract  or  transaction  ...  without  the 
use  of  competitive  procedures, ...  if ...  competitive  procedures  were  used  for  the 
selection  ...  in  the  transaction;  and  the  participants....  successfully  completed  the 
prototype  project ..." 

Note  that  the  authority  to  enter  into  prototype  project  emphasizes  relevance  to  mission.  It  does  not 
provide  legal  definition  of  "prototype."  It  does  not  imply  that  "prototype  projects"  refer  exclusively  to 
new  technology.  Rather,  the  implication  is  that,  consistent  with  common  use  of  the  term  in  the 
engineering  community,  a  "prototype"  is  simply  a  design  model.  A  design  model  can  be  physical  or 
virtual.  It  can  address  technology  or  process.  Therefore,  a  "prototyping  project"  might  aim  to  develop 
or  select  prototype(s),  or  use  prototype(s)  to  evaluate  alternatives.  Significantly,  prototyping  projects 
might  be  used  to  perform  market  research,  material  solution  analysis,  and  trades  studies.  Prototyping 
is  specifically  included  in  "Technology  Maturation  and  Risk  Reduction,"  and  "Engineering  and 
Manufacturing  Development"  phases  of  "The  Defense  Acquisition  System",  per  DoDI  5000.01.  A 
prototyping  project,  i.e.  operational  evaluation  of  a  mature  prototype,  might  take  the  place  of  the 
"Limited  Rate  Initial  Production"  phase. 

Use  of  OTA  for  prototyping  projects  with  value  less  than  $50M  need  not  be  justified  in  writing.  For  use 
of  OTA  for  projects  above  that  threshold,  the  Senior  Acquisition  Executive  (SAE)  must  determine  and 
find  (D&F)  in  writing  that  some  aspect  of  the  OTA  approach  is  essential  to  the  success  of  the  project. 

E.g.  if  greater  speed-to-capability,  or  outreach  to  non-typical  vendors,  is  essential,  and  not  likely  to  occur 
through  the  traditional  approach,  the  SAE  might  determine  that  OTA  is  essential. 

Intent  is  that  performers  who  competitively  earn  the  right  to  deliver  design  models,  have,  upon 
satisfactory  validation  and  verification  of  the  design  model,  earned  the  right  to  deliver  production  units 
of  the  design,  a  priori,  i.e.  without  further  competition. 

A  summary  of  the  applicability  of  OTA  follows: 

•  "Other"  means  "other  than  a  'contract'  or  'grant'",  and  therefore  not  subject  to  Federal 
Acquisition  Regulations  (FAR) 

•  Used  when  flexibility  in  procurement  agreement  is  paramount  --  e.g.  when  application  of  FAR 
parts  is  not  feasible  or  appropriate  to  achieve  legitimate  government  objectives  --  especially 
regarding  innovative  objectives 
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•  OTA  is  flexible,  for  example: 

•  FAR  Cost  Accounting  Standards  need  not  apply 

•  FAR  based  competitive  procedures  need  not  apply 

•  FAR  standards  for  intellectual  property  rights  need  not  apply 

•  The  US  Government  (USG)  may  execute  a  transaction  for  a  prototyping  project  with  a  single 
performer  or  a  team  of  performers. 

•  USG  may  also  execute  an  "umbrella"  transaction  with  an  "open"  consortium  of  potential 
project  performers/ 

•  Under  10  USC  2371,  purpose  is  basic  or  applied,  research 

•  Under  10  USC  10  2371b,  purpose  is  to  conduct  prototype  projects,  i.e.  evaluation  of  design 
models 

•  If  an  OT  for  prototype  projects  is  awarded  under  competitive  procedures,  and  the  project  is 
successful,  then  the  Government  may  make  direct  award  for  follow  on  OTs  for  prototype 
projects,  or  for  OTs  or  contracts  for  production. 
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Rationale  for  Executing  "Other  Transactions"  (OT) 

Why  OT? 

OTA  may  be  applied  to  conduct  basic  and  applied  research,  and  to  conduct  prototyping  projects.  The 
term  "prototype"  in  not  defined  by  statute,  but  is  generally  interpreted  to  mean  any  virtual  or  physical 
design  model  of  a  technology  or  process  used  to  evaluate  engineering  solutions.  Thus  prototyping 
projects  may  include  studies,  invention,  modeling,  simulations,  purchases  of  commercial  solutions,  test, 
and/or  evaluation  that  aim  to  help  evaluate  engineering  solutions,  and  execute  associated 
programmatic  decisions.  That  prototyping  activity  might  occur  during  any  phase  of  the  Defense 
Acquisition  System. 

If  an  OT  for  a  prototype  project  is  awarded  under  competitive  procedures,  and  the  project  is  successful, 
then  the  government  may  make  a  direct  award  of  a  contract  or  transaction  for  follow-on  production  and 
distribution  without  further  competition.  Under  that  authority,  a  project  office  can  efficiently  manage 
competition  based  on  robustly  demonstrated  performance  with  respect  to  cost,  schedule,  and  technical 
parameters.  Having  achieved  prototyping  exit  criteria,  the  government  can  directly  transition  the 
developed,  or  demonstrated,  capability,  as  either  a  COTS  product,  or  a  purpose-built  military  system. 

The  following  requirements  and  strategies,  for  example,  might  justify  use  of  OTA: 

•  Conduct  open  ended  basic  and  applied  research  with  elite  commercial  and/or  academic 
partners 

•  Conduct  Research,  Development,  Test  &  Evaluation  (RDT&E),  certification,  and  then  transition 
perishable  technology  to  manufacturing  and  distribution  before  it  is  superseded  by  the  next 
generation 

•  Crowdsource  requirements  to  broad  and  unknown  market  of  innovative  solution  providers 

•  Establish  especially  close  government-industry  partnership  with  trusted  firms 

•  Equitably  share  developmental  risk  with  industrial  partners 

•  Conduct  adaptive,  evolutionary  S&T  and  RDT&E,  i.e.  each  preceding  discovery  impacts 
subsequent  procurement  approach 

•  Evaluate  alternative  procurement  governance  models  for  potential  incorporation  into  the  FAR 

•  Employ  non-traditional  incentive  models,  including  with  respect  to  IPR,  not  feasible  under  the 
FAR,  for  any  of  the  above,  or  any  other  legitimate  purpose 


Why  an  OT  with  an  open  consortium  instead  of  a  single  industry  partner? 

The  USG  may  execute  an  OT  for  prototyping  projects  with  an  open  consortium.  The  purpose  of  this 
arrangement  is  to  establish  a  marketplace  of  performers  who  can  efficiently  and  effectively  compete 
and  collaborate  to  perform  in-scope  prototyping  projects  with  funding  awarded  by  the  USG.  An  "open 
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consortium"  in  this  sense  is  any  evolving  federation  of  potential  performers,  with  low  barriers  of  entry 
to  all  comers,  the  purpose  of  which  is  to  execute  the  requisite  portfolio  of  prototyping  projects.  The 
consortium  is,  by  definition,  the  sum  of  its  members.  As  a  practical  matter,  OT  consortia  usually  have 
consortium  managers  empowered  to  enter  into  legally  binding  agreements,  and  to  govern  consortium 
activity.  Typically  professional  consortium  managers  create  the  requisite  consortium,  or  adapt  a 
consortium  that  exists  for  another  purpose,  in  response  to  USG  solicitation.  They  might  do  that  on  their 
own  initiative,  or  at  the  request  of  potential  consortium  members.  The  USG  awards  an  "umbrella"  OT 
for  prototype  projects--  with  specified  funding  ceiling  and  period  of  performance  -to  a  consortium.  The 
USG  may  then  efficiently  solicit  and  award  individual  prototype  projects  to  consortium  members,  using 
streamlined  competitive  procedures  agreed  under  terms  of  consortium  membership. 

The  Army  Contracting  Command  invented  consortium-type  OTs  and  pioneered  their  use.  Their  many 
years'  experience  verifies  the  following  good  outcomes: 

•  Low  barriers  to  entry,  simplified  procedures,  and  clear  incentives,  indeed,  attract  innovative 
non-traditional  performers. 

•  Consortia  provide  venues  for  competition  among  members  that  are  consistent  with  FAR  "free 
and  open"  intentions,  low  maintenance  for  the  government,  and  efficient  both  for  government 
and  industry  participants. 

•  The  same  venue  that  provides  for  competition  is  equally  efficient  for  catalyzing  partnerships 
among  members. 

•  Solicitation-to-award  timelines  are  much  shorter  than  typical  under  the  FAR. 

What  are  risks  and  mitigations? 

Consider  the  following  characterization  of  relative  risks  and  rewards  of  procurement  via  OTA  vs. 
traditional  FAR  approaches. 

Risk:  Bureaucracy  and  rigidity  of  typical  FAR-based  contracting  exacerbates  the  risk  that  programs  will 
not  achieve  cost,  performance,  and/or  schedule  targets. 

Discussion:  Multiple  watchdog  report  and  policy  initiatives  acknowledge  that  weapon  system 
acquisition  is  not  sufficiently  efficient  or  effective.  The  existence  of  this  deficiency  in  the 
current  acquisition  process  proves  that  the  status  quo  approach  to  managing  procurement  risk 
is  not  adequate. 

Mitigation:  Use  OTA  to  experiment  with  alternative  procurement  language  to  find  best  practices 
for  achieving  cost,  performance,  and  schedule  targets  associated  with  harvesting  value  from  the 
highly  volatile  commercial  technology  marketplace.  As  best  practices  are  identified,  apply  them 
to  develop  new  standard  FAR-based  contracting  language. 

Risk:  Flexibility  of  OTA  will  lead  to  abuse. 

Risk:  Non-standard  procurement  language  used  in  OTAs  will  lead  to  binding  agreements,  e.g.  re 
intellectual  property  rights,  that  are  unfavorable  to  the  government. 

Discussion:  The  following  data  points  are  relevant: 
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•  10  USC  2371b  is  basis  of  OTA  for  Prototyping  Projects  for  DoD.  The  OSD  2017  OT  Guide 

for  Prototyping  Projects  explains  uses  and  constraints  in  context  with  OSD  policy. 
Generally: 

o  Senior  Acquisition  Executives  (SAE)  of  DoD  departments  and  agencies  may 
authorize  OTs  of  $50M  -  $250M,  and  may  delegate  authorization  authority  for 
OTs  of  less  than  $50M.  The  OSD  SAE  may  authorize  OTs  for  larger  amounts. 
Authorization  must  explain  why  use  of  OTA  is  essential  to  project  objectives, 
and  be  in  writing. 

o  Typically,  commercial  partners  must  pay  1/3  development  costs.  That 

requirement  may  be  waived  if  the  commercial  team  includes  small  business(es) 
or  non-traditional  Defense  contractors.  Non-traditional  Defense  contractors  are 
defined  as  companies  that  have  not  entered  into  procurement  agreements  with 
DoD  wherein  they  were  required  to  fully  comply  with  FAR  Cost  Accounting 
Standards  (CAS)  within  the  last  twelve  month  period. 

o  The  term  "prototype"  is  not  defined  by  statute.  Generally,  it  refers  to  how  a 
thing  is  used,  not  what  it  is.  "Prototypes"  are  design  models  used  to  evaluate 
potential  solutions.  They  may  be  physical  or  virtual,  and  may  address 
technology  or  processes. 

o  By  their  nature  prototyping  projects  fundamentally  address  Test  and/or 
Evaluation.  Hence,  normally  use  of  RDT&E  funds  is  appropriate.  However, 
according  to  the  Financial  Management  Regulation,  there  are  some  exceptions. 
See  Appendix  C-l  for  detail. 

o  In  addition  to  executing  OTs  with  individual  firms  and  teams,  DoD  departments 
have  executed  OTs  with  open  consortia.  The  term  "open"  means  that  barriers 
to  membership  and  dialog  between  government  and  industry  are  low.  When  an 
"umbrella"  consortium-type  OT  is  in  place  -with  established  funding  ceiling  and 
period  of  performance  —  transactions  for  individual  projects  may  be  solicited 
and  awarded  very  quickly.  Thus,  consortium-type  OTs  can  effectively  establish  a 
marketplace  around  government  requirements. 

o  If  an  OT  for  prototyping  has  been  established  under  competitive  procedures, 
and  the  prototyping  project  is  deemed  "successful",  the  Government  may  make 
a  direct  award  of  a  traditional  contract,  or  an  OT,  for  production.  Further 
competition  is  not  required. 


•  A  Rand  assessment  of  OTA  circa  2000,  based  on  a  sample  of  21  of  a  total  of  72  DoD 
projects  executed  under  OTA,  concluded  that  rewards  outweighed  risks.  This  is 
particularly  true  under  certain  circumstances  including  projects  "offering  DoD  the 
opportunity  to  benefit  from  innovative  business  relationships." 


Chris  Gunderson/SAF/AQ  OTI/(o)703-693-4177/(m)831-224-5182/25July2017 


23 


•  2001  GAO  Congressional  Testimony  concludes  that  OTAs  are  effective  tools  for  helping 
DoD  leverage  commercial  innovation,  that  DoD  has  taken  important  steps  to  provide 
guidance  for  use  of  these  tools,  but  that  follow-on  training  is  essential.  The  report 
describes  mixed  results  against  objectives,  but  makes  no  reference  to  suspected  or 
documented  abuse. 

•  A  2006  GAO  report  on  DoD  vulnerabilities  to  fraud  waste  and  abuse,  performed  in 
response  to  the  Darleen  Druyun  scandal,  cites  interagency  contractual  agreements,  and 
IDIQ  contracts  as  risky,  but  does  not  mention  OTA. 

•  A  2012  Rand  report  of  lessons  learned  from  the  Future  Combat  System  (FCS)  failure 
noted  "use  of  OTA  in  the  design  concept  phase  was  clearly  warranted."  The  report  also 
documented  that  although  rationale  for  use  of  OTA  for  later  program  phases  was 
questionable,  and  Congress  was  concerned  over  potential  for  abuse,  the  choice  of  OTA 
rather  than  FAR-based  contracts  was  not  the  central  issue.  (Note  that  no  actual  abuse 
was  documented.)  The  central  issue  was  not  effectively  aligning  fiscal  incentives  with 
clear  value-based  programmatic  outcomes. 

•  A  2012  GAO  report  of  DHS  use  of  OTA  concludes  that  while  DHS  finds  OTA  to  be  an 
important  tool  with  enough  flexibility  to  allow  development  of  critical  technology,  DHS 
metrics  regarding  OTA  effectiveness,  and  audit  methods  in  general,  are  inadequate.  The 
report  cites  DoD's  metrics  and  audits  as  an  example  of  a  better  process.  In  this  review 
of  58  OTAs  over  eight  years,  the  report  explains  risk  of  abuse,  but  does  not  mention  any 
issues  associated  with  documented  or  suspected  abuse. 

•  A  2016  GAO  report  explains  how  OTA  is  employed  by  eleven  different  federal  agencies. 

•  A  2013  article  in  "The  Procurement  Lawyer"  Journal  suggests  that  given  tightening 
Defense  budgets,  and  reluctance  of  firms  in  the  technology  sectors  to  enter  in  FAR- 
based  contracts,  OTAs  provide  a  viable  alternative  means  for  DoD  to  leverage  industrial 
innovation.  The  article  explains  legal  basis  and  rationale  for  use  of  OTA,  and  provides 
useful  cautions. 

•  Conversation  with  Denise  Scott,  RDECOM-ARDEC  Legal  Office,  (circa  Feb  2017)  indicates 
that  the  consortium-type  OTAs  she  has  overseen  for  ten  years,  in  partnership  with  Army 
Contracting  Command  (ACC),  have  been  low  maintenance  from  a  legal  perspective.  In 
particular,  competing  vendors  have  not  protested  a  single  award.  Meanwhile,  according 
Kim  Blancuzzi  of  the  Army  Weapon  System  IT  Center,  technical  overseer  of  the  ACC  C5 
Technologies  OTA,  typical  award  time  is  less  than  two  months. 
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•  The  Air  Force  Research  Laboratory,  Rome  NY,  established  the  Open  System  Acquisition 
Initiative  OT  with  the  System  of  System  Consortium  (SOSEC)  mid  FY16  with  $100M 
ceiling  and  5  year  period  of  performance.  This  is  the  first  USAF  consortium-type  OT. 
After  less  than  one  year's  effort  approximately  %  of  the  ceiling  was  executed  across  a 
dozen  or  so  projects,  the  consortium  membership  has  grown  continuously  (now  ~200 
firms),  and  solicitation  to  award  times  are  measured  in  weeks. 


Mitigation:  Clearly  the  risks  associated  with  OTA  flexibility  are  real.  Further,  lack  of  objectives 
statistics  regarding  historical  OTA  effectiveness  is  troubling.  However,  lack  of  clear  objective 
statistics  regarding  FAR-based  contracting  effectiveness  is  equally  troubling.  To  address  these 
issues,  the  following  risk  mitigating  actions  are  appropriate. 

•  Learn  lessons  by  working  closely  with  procurement  professionals  who  have  successfully 
managed  risks  and  harvested  rewards  associated  with  consortium-type  OTs. 

•  Build  a  Systems  Engineering  Plan  the  breaks  the  project  into  relatively  small  parallel 
efforts.  Use  OTA  to  issue  small  focused  awards  with  rapid  turnaround  cycles.  "Failures" 
will  occur  fast  and  cheap  when  they  occur.  Turn  so-called  failures  into  lessons  learned 
for  the  next  iteration. 

•  Handpick  a  team  of  procurement  professionals  to  manage  the  OT  process.  Provide 
sufficient  resources  to  allow  them  to  succeed. 

•  Establish  engaged  oversight  at  the  appropriate  level,  commensurate  with  dollar  value 
and  criticality  of  projects. 

•  Define  objective  adaptive  acquisition  metrics  for  validating  and  verifying  requirements 
and  tracking  project  cost,  performance,  and  schedule.  Use  robust  "plug  testing"  to 
collect  those  metrics  and  populate  a  risk/reward  dashboard  for  review  at  all  levels. 
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Senior  Acquisition  Executive  (SAE)  OTA  Decision  and  Finding  (D&F) 
Template  (USAF) 


Again,  use  of  OTA  for  prototype  projects  with  value  between  $50  and  $250  million  require  written 
justification  by  the  appropriate  service  SAE.  Projects  with  greater  value  require  approval  by  OSD. 
Consider  the  following  template  for  written  D&F. 

SAF/AQ  Letterhead 

MEMORANDUM  FOR _ (Contracting  Office  that  will  Execute  OT) 

FROM:  (For  example)  SAF/AQ 
1060  Air  Force  Pentagon 
Washington  DC  20330 

SUBJECT:  Written  Determination  to  Use  an  Other  Transaction  for  Prototype  Project  Exceeding  $50,000, 
but  not  Exceeding  $250,000  -  " _ "  (Name  of  Vehicle) 

1.  Pursuant  to  Section  845  of  Public  Law  103-160,  as  amended,  I  have  determined  that 

_ (Organization  requesting  the  D&F)  may  execute  the _ (Name  of 

Vehicle)  as  a  prototype  project  using  "Other  Transactions"  Authority  (OTA)  under  10  U.S.C.  2371. 

2.  _ 's  (Organization's)  proposed  approach  to _ (Name  of  Vehicle) 

represents  a  unique  methodology  for  obtaining  capabilities  directly  relevant  to  mission  effectiveness  of 
military  personnel  and  their  supporting  platforms,  systems,  components,  and/or  materials.  Relevant 
non-traditional  and  commercial  organizations  have  expressed  concern  with  cost  accounting  standards, 
intellectual  property  rules,  and  audit  requirements  of  Government  contracting  and  therefore  refrain 
from  participating  in  in  traditional  federal  procurement  opportunities.  The  Air  Force  is  not  able  to  take 
sufficient  advantage  of  the  research  and  development  conducted,  and  the  existing  products,  services, 
and  processes  offered  by  these  organizations.  This  research  and  development,  and  these  products  and 
services,  would  likely  add  significant  benefit  to  weapon  system  development  and  sustainment.  Further, 
in  some  cases  the  technology  involved  is  highly  perishable,  and  traditional  application  of  FAR  process 
would  likely  preclude  its  timely  deployment.  Judicious  application  of  OTA  for  prototyping  projects  (OTP) 
provides  an  effective  means  to  alleviate  these  issues,  and  therefore  will  catalyze  significant 
enhancement  to  warfighting  capability. 

3.  The  OTP  negotiated  as  part  of  the _ (Name  of  Vehicle)  acquisition  shall  be  limited 

to  a  total  value  of  not  more  than _ (no  more  than  $250M)  (including  all  options). 

Furthermore,  non-governmental  awardees  for  projects  will  either  include  at  least  one  nontraditional 
Defense  contractor,  or  small  business,  participating  to  a  significant  extent,  or  will  contribute  at  least  one 
third  of  project  costs. 
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4.  I  have  found,  as  further  provided  in  the  attached  background  paper  (explanation  of  the  purpose  and 
rationale  for  the  requested  vehicle  provided  by  requesting  organization)  that:  (a)  the 

_ (Name  of  Vehicle)  meets  the  requirements  of  Section  845(a)(2)(A)(i)  and  (ii)  of 

P.L.  103-160,  as  amended;  and  (b)  such  authority  is  essential,  in  view  of  the  need  to  attract  non- 
traditional  Defense  contractors,  and  to  achieve  sufficient  speed-to-capability  by  promoting  the  success 

of _ prototype  projects;  and  (c)  exceptional  circumstances  justify  the  use  of  OTA 

to  provide  for  innovative  business  arrangement  or  structures,  notably  use  of  a  consortium,  that  would 
not  be  feasible  or  appropriate  under  a  contract.  Therefore,  I  authorize  execution  of  the 

_ (Name  of  Vehicle)  under  OTA  in  accordance  with  Section  845(a)(2)(A)  of  P.L.  103- 

160,  as  amended. 

Signature  Block 
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Considerations  for  Establishing  and  Managing  "Other  Transactions"  (OT) 
Consortia 


The  following  guidance  is  derived  from  lessons  learned  by  actual  practitioners. 


Prime  Directives 

1.  Leverage  the  opportunity  to  innovate  that  exists  because  OTs  need  not  comply  with  FAR. 

2.  Notwithstanding#!  above: 

a.  Do  not  reinvent  existing  contractual  clauses,  or  sub  clauses,  that  work  just  fine. 

b.  If  you  do  use  FAR  methods  and  clauses,  consider  re-naming  them  to  preclude  invoking  the 
kind  of  legacy  thinking  you  are  trying  to  supersede. 

3.  Plan  to  adjust  terms  of  agreements  early  and  often  as  lessons  are  learned;  broadcast  those  intentions 
as  widely  as  possible  at  every  opportunity. 

Humans  tend  to  default  to  the  familiar.  Resist!  As  is  the  case  in  any  engineering  endeavor,  the 
appropriate  first  step  for  any  OTA-related  project  is  to  clearly  specify  desired  outcomes  for  both 
governance  processes  and  delivered  capability.  Next,  determine  how  best  to  incentivize  those 
outcomes  given  facts-of-life  boundary  conditions.  If  an  existing  set  of  contractual  clauses  or  sub  clauses 
—  either  in  government  or  industrial  boilerplate  —  serve  the  purpose,  re-use  those  elements,  but  only 
those  elements.  (Consider  using  different  words  to  express  the  legacy  elements!)  If  and  where 
adequate  boiler  plate  procurement  artifacts  do  not  exist,  invent  new  clauses  with  the  intent  of  reusing 
them  in  the  future.  Regardless,  accept  that  whatever  agreements  you  make  will  not  be  perfect.  Plan 
from  the  start  to  adjust  downstream.  Make  sure  OT  clauses  address  the  need  for  government  and 
industry  to  continuously  evaluate  progress,  and  adjust  details  of  the  agreement  as  necessary. 

Nature  of  "Transactions"  in  a  Consortium  Arrangement 

There  are  essentially  two  kinds  of  transactions  that  occur  between  government  and  industry  in  an  OT 
consortium:  an  "umbrella"  transaction  between  the  USG  and  the  consortium  governing  body,  and 
prototyping  project  transactions  between  the  USG  and  actual  performers.  The  umbrella  transaction 
establishes  the  terms  of  reference  for  the  envisioned  prototyping  activity.  It  will  describe  scope,  ceiling, 
and  period  of  performance  for  the  entire  envisioned  body  of  work.  It  will  also  specify  the  services  to  be 
provided  by  the  consortium  governance  authority  -which  is  typically  a  consortium  manager  -  and  the 
associated  compensation.  Usually  compensation  for  the  consortium  manager  does  not  occur  until 
funds  are  awarded  for  individual  projects.  In  that  case,  executing  the  umbrella  transaction  does  not 
require  transfer  of  funds.  Alternatively,  the  umbrella  transaction  could  include  delivery  of  a  retainer  fee 
to  the  CM. 
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Once  an  umbrella  transaction  is  in  place,  transactions  for  individual  projects  may  be  executed.  These 
project  transactions  are  legally  binding  procurement  instruments  between  the  USG  and  the  actual 
performing  firm.  The  consortium  manager  is  not  a  party  to  these  transactions,  but  will  usually  serve  as 
the  broker.  I.e.  the  USG  sends  money  to  the  CM.  The  CM  takes  the  appropriate  fee  for  service  per 
terms  of  the  umbrella  transaction,  and  delivers  the  remaining  funds  to  the  performers  per  terms  of  the 
project  transaction. 

Ceilings  and  Scope  of  "Other  Transactions" 

Service  Secretaries  have  authority  to  establish  "Other  Transactions"  (OT)  for  prototyping  with  awards 
between  $50  and  $250M.  Regarding  consortium  type  OTs,  these  award  values  have  typically  been 
interpreted  as  a  cumulative  ceiling  for  all  prototype  projects  executed  under  the  OTA  across  a  specified 
period  of  performance.  Presumably,  the  Senior  Acquisition  Executive  (SAE)  will  refresh  the  ceiling  when 
circumstances  dictate.  It  is  also  possible  that  the  SAE  might  agree  that  the  award  limits  apply  to  each 
project  executed  under  the  "umbrella"  of  a  consortium-type  OT. 

Regardless,  the  scope  of  the  OT  will  influence  the  amount  and  type  of  fiscal  activity  that  occurs,  and  how 
quickly  the  cumulative  award  ceiling  is  reached.  The  scope  will  also  influence  how  and  whether  the  OTA 
catalyzes  acquisition  across  program  lines,  which  may  or  may  not  be  an  objective.  Thus,  the  scope  will 
influence  the  numbers  and  expertise  of  members  of  the  contract  office  required  to  execute  and  monitor 
OTAs. 

Per  all  the  above,  specifying  the  scope  of  a  consortium  type  OT  is  both  important  and  potentially 
difficult.  Getting  the  right  mix  of  manageable  activity  and  innovative  outcomes  will  require  some  trial 
and  error.  Consider  options  for  provisioning  flex  and  surge  support  by  procurement  experts  for  OTs,  just 
as  for  other  omnibus-type  vehicles. 

OT  Consortium  Source  Selection  Criteria 

Government  stakeholders  should  design  the  target  capabilities  and  demographics  of  both  the 
consortium  members,  and  the  consortium  management  structure. 

Consortium  members  may  represent  different  technology  sectors,  different  sizes,  and  differing  degrees 
of  familiarity  with  government  procurement  process.  They  may  provide  particular  managerial, 
engineering,  and/or  analytical  capabilities. 

The  OT  consortium  itself  is,  by  definition,  the  sum  of  its  member.  As  such,  the  consortium,  per  se, 
should  be  a  not-for-profit  organization  that  exists  to  equitably  serve  government  and  private  interests  in 
performance  of  the  USG  mission  at  hand.  However,  OT  consortia  need  not  be  formally  incorporated  as 
not-for-profit  companies.  They  need  not  be  legally  incorporated  at  all.  However,  an  OT  consortium,  i.e. 
the  members  of  the  consortium,  should  be  bound  by  some  governing  charter  that  specifies  alignment 
with  the  intention  and  scope  of  the  government  requirements.  Ideally  the  OT  consortium  membership 
process  presents  low  barriers  to  joining  on  the  one  hand,  but  also  provides  a  reasonable  degree  of 
vetting  to  assure  good  standing  in  context  with  the  consortium  charter  on  the  other. 
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Theoretically,  the  members  of  an  OT  consortium  could  agree  on  some  management  process  that  does 
not  require  a  single  point  of  contact  to  act  on  behalf  of  all  the  members.  It  is  much  more  likely  that  a 
Consortium  Manager  will  fill  that  role.  Indeed,  very  often,  professional  consortium  managers  will 
respond  to  USG  solicitations  by  causing  OT  consortia  to  form  in  the  first  place.  Or,  if  a  group  of 
companies  wishes  to  respond  to  an  OT  solicitation,  the  group  will  likely  seek  the  services  of  a 
professional  consortium  manger  to  organize  their  bid. 

Consortium  Managers  can  be  individuals,  for  profit  corporations,  or  not-for-profit  corporations.  The 
CM  serves  as  an  independent  agent  of  the  government,  and  will  generally  not  be  eligible  to  compete  for 
project  awards.  Some  argue  that  retaining  a  501. c. 3  not-for-profit  firm  for  this  function  reduces 
potential  for  perceived  conflict  of  interest. 

CM's  should  demonstrate  ability  in  at  least  three  categories:  1)  brokering  government  requirements 
across  its  membership;  2)  managing  and  documenting  fiducial  activities  necessary  to  transfer  funds  from 
government  to  consortium;  3)  provisioning  value  added  services  with  respect  to  the  government's 
objectives.  Examples  of  value  added  services  include  targeted  recruiting  of  new  members,  facilitating 
acquisition  process  innovation,  performing/facilitating  technical  validation  and  verification,  capturing 
evolving  best  practices  and  standards,  managing  events,  performing  training,  etc.  Government 
stakeholders  should  consider  how  they  want  the  CM  to  facilitate  government  objectives,  and  if  and  how 
the  CM  might  potentially  augment  governmental  activity. 

Accordingly,  the  government's  Statement  of  Objectives  (SOO)  supporting  the  solicitation  and  award 
might  address  some  or  all  of  the  following  topics.  The  SOO  should  also  specify  objective  measures 
and/or  subjective  evaluation  factors  associated  with  each  topic. 

•  Number  of  members  who  are  "traditional,"  non-traditional,  large,  small,  foreign  domestic 

•  Technology  sectors  represented  by  members 

•  Relevant  services  represented  by  members,  e.g.  engineering,  program  management,  analysis, 
test  &  evaluation 

•  CM  financial  expertise 

•  CM  technological/engineering  expertise 

•  CM  acquisition  expertise 

•  CM  training  expertise 

•  Prior  performance  by  members'  and  or  CM  regarding  the  scope  of  the  OTA. 

•  Method  for  firewalling  consortium  manager  from  actual  prototype  project  execution,  e.g.  use  of 
501. c. 3. 

Consortium  Managers'  Fee  Structure 

The  government  will  consider  multiple  potential  fee  structures  in  the  management  of  this  consortium.  A 
fixed  fee  per  transaction  is  the  most  straightforward  approach.  However,  the  benefits  of  alternative 
structures  might  outweigh  the  value  of  simplicity.  Ultimately,  the  government  is  seeking  a  fee  structure 
that  helps  minimize  expense  to  the  government  while  maximizing  competition  within  the  consortium. 
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The  government  should  consider  a  range  of  transaction  fee  structures,  as  well  as  incentive-based 
approaches  that  may  encourage  the  CM  to  perform  some  or  all  of  the  value-added  services  discussed  in 
the  previous  section,  for  example  expanding  the  bidding  pool  to  an  optimal  number  of  participating 
vendors  (particularly  non-traditional  defense  contractors). 

Some  examples  of  potential  transaction  fee  structures  include: 

1.  Fixed  transaction  fee  (same  fee  regardless  of  price  of  project) 

2.  Tiered  transaction  fees  (fees  vary  for  projects  that  fall  within  different  price  ranges)  based  on  project 

price,  and/or  services  rendered  by  CM,  adjusted  to  exclude  hardware,  material,  and  travel  costs1 

3.  Tiered  transaction  fees  based  on  total  project  price 

4.  Percentage  of  price  up  to  a  maximum  $  amount  based  on  project  price  adjusted  to  exclude  hardware, 

materials,  and  travel  costs 

5.  Percentage  of  project  price  up  to  a  maximum  $  amount 

Proposed  incentive  structures  could  be  evaluated  as  part  of  an  offeror's  bid.  It  might  be  sufficient  to 
explain  how  the  proposed  transaction  fee  approach  incentivizes  keeping  the  cost  of  individual  efforts 
down  in  order  to  create  greater  opportunity  for  further  transactions  under  the  ceiling  of  the  agreement. 
Government  could  also  consider  how  the  transaction  fee  might  further  other  government  objectives. 

For  example  the  fee  structure  might  incentivize  higher  quality  competition  during  PlugTesting,  since  the 
CM  will  be  compensated  based  on  the  number  of  PlugTest  participants  that  receive  an  award. 

The  government's  expectation  might  be  that  fixed  fee  plus  incentive  structures  would  result  in  a  lower 
fixed  fee  per  transaction  than  would  occur  without  the  incentive.  Offerors  who  are  interested  in  this 
structure  should  define  the  cost  differences  between  the  two. 

The  government  will  consider  alternate  fee  structures  to  the  examples  provided  above.  Flowever, 
offerors  should  indicate  how  these  proposed  fee  structures  can  be  practically  implemented  and 
minimize  cost  to  the  government  and  maximize  competition  for  each  transaction. 

Intellectual  Property 

Following  award  of  the  agreement,  the  government  will  work  with  the  Consortium  Manager  to  establish 
a  set  of  Intellectual  Property  templates  that  define  varying  degrees  of  government  rights  to  products 


JThe  exclusion  of  hardware,  materials,  and  travel  from  the  fee  calculation  is  intended  to  capture  labor 
costs  as  the  main  indicator  of  risk  in  a  given  project,  which  may  lead  to  more  challenging  management 
during  performance  of  a  given  project.  Projects  that  consist  in  large  part  of  material,  hardware,  and/or 
travel  are  perceived  to  be  less  risky  and  require  less  potential  for  substantial  involvement  on  the  part  of 
the  Consortium  Manager 
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acquired  through  the  OT  Consortium.  Each  government  program  that  utilizes  the  OT  Consortium  will 
select  the  appropriate  template  to  meet  its  needs,  with  input  from  the  Consortium  Manager. 
Government  programs  may  choose  to  define  Intellectual  Property  clauses  separate  from  the  pre-defined 
templates  on  a  project-by-  project  basis,  in  consultation  with  the  Consortium  Manger. 

Project  Award  Fee  Structure 

Significantly,  OTAs  are  not  constrained  by  FAR  fee  structures,  intellectual  property  models,  or  cost 
accounting  standards.  Government  stakeholders  and  CMs  need  not  default  to  familiar  FAR  language. 
Rather,  they  may  first  determine  the  most  effective  way  to  incentivize  the  targeted  outcome.  Then  align 
the  fee  structure  in  a  way  that  provides  the  appropriate  incentive.  If  there  is  any  new  development 
involved,  likely  the  right  fee  structure  should  be  some  form  of  best  effort  or  incentive-over-cost  model. 
Given  the  option  for  price  sharing  between  government  and  industry  the  "plus"  might  actually  be  a 
negative  number  in  terms  of  the  industrial  contribution.  The  "incentive"  might  take  the  form  of  a 
lucrative  intellectual  property  rights  agreement.  Most  likely,  fixed  price  models  should  only  apply  when 
the  project  demands  simple  delivery  of  existing  COTs  products  and  services. 
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Solicitation  for  Creation  of  an  OT  Consortium 


Please  consider  the  following  notional  example  of  an  OT  consortium  solicitation  as  a  potential  template. 
Notice  Type:  Solicitation 

Added: _ 

Title: _ 

Synopsis: 

XYZ  Program  Office  (XPO)  is  releasing  this  notice  to  inform  interested  parties  of,  and  seek 
comments  and  questions  concerning,  a  potential  forthcoming  award  of  a  transaction  for 
establishing  a  consortium  for  conducting  prototyping  projects  per  10  USC  2371b:  "Other 
Transactions"  Authority  for  Prototyping  Projects. 

XPO  is  exploring  options  regarding  competitive  award  of  an  "other  transaction"  (OT)  to  an 
eligible  new  or  existing  not-for-profit  consortium  of  large  and  small  organizations  representing 
traditional  and  non-traditional  Defense  contractors,  as  well  as  academic  institutions.  Consortia 
need  not  be  formally  incorporated,  but  they  may  be.  The  mission  of  the  consortium  must 
include  performing  research,  development,  test  and  evaluation  within  prototyping  projects  that 
address  XYZ  and  customer  requirements  for  X  systems. 

XPO  may  choose  to  award  an  umbrella  transaction  for  "Adaptive  Acquisition  for  X  Systems" 
(A2XS).  XPO  would  make  such  an  award  to  an  eligible  consortium  internal  governing  body,  or 
its  designated  third  party  manager.  Consortium  managers  (CM)  or  governing  bodies  must  have 
means  and  legal  authority  to  represent  the  interest  of  all  the  members  of  the  organization, 
organize  their  activities,  and  make  binding  agreements  on  their  behalf. 

After  establishing  an  "umbrella"  OT  with  a  consortium  or  its  designated  consortium  manager, 
the  USG  would  consider  making  solicitations  and  awarding  funded  transactions  for  an  indefinite 
number  of  prototyping  projects  within  the  scope  and  period  of  performance  of  the  umbrella 
transaction.  It  is  likely  that  the  CM,  if  any,  would  receive  a  portion  of  these  awards  as  a 
service  fee.  However,  the  CM  must  remain  independent  of  the  internal  competitive  process, 
and  may  not  participate  as  a  participant  in  the  prototyping  project  transactions  per  se.  Any 
alternative  governance  structure  must  explicitly  explain  how  it  avoids  conflict  of  interest  in  the 
internal  competitive  process. 
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Scope  and  Objectives: 


XPO  has  found  that  its  legacy  acquisition  methods  tend  to  be  inflexible  and  overly  bureaucratic, 
leading  too  often  to  unsatisfactory  efficiency  and  effectiveness  with  respect  to  cost,  schedule, 
and  system  and  process  performance.  Therefore  XPO  aims  to  take  advantage  of  the  inherent 
flexibility  afforded  by  "other  transactions"  authority  to  achieve  the  following  general  objectives. 

Procurement  Objectives 

•  Expose  USAF  requirements  to,  and  lower  barriers  for,  broad  community  of 
traditional  and  non-traditional  solution  providers. 

•  Accelerate  solicitation  to  award  timeline. 

•  Employ  objective,  test-based.  Validation  and  Verification  as  basis  of  Analysis  of 
Alternatives  (AoA),  trades  analysis,  source  selection,  and  performance  monitoring. 

•  Develop  Intellectual  Property  Regimes  (IPR)  and  Data  Rights  that  incentivize 
mutually  beneficial  government-industry  partnership. 

•  Transition  successfully  tested  prototype  applications  as  vendor-supported  off-the- 
shelf  products. 

Management  and  Engineering  Objectives 

•  Provision  continuous  feedback  loop  with  operational  customers. 

•  Establish  persistent,  broadly  accessible,  test  tools,  and  execute  test  cases  to  achieve 
Validation  and  Verification  (V&V)  in  support  of:  market  analysis,  AoA,  trades, 
development,  operational  transition,  certification,  and  life  cycle  tech  refresh  as 
parallel  activities. 

•  Minimize  risk  of  technology  perishability  by  executing  rapid,  iterative,  evolutionary, 
demonstration-to-delivery  cycles. 

•  Continuously  evolve  an  appropriately  "open"  technical  architecture  that  optimizes 
government  investment  in  both  leveraging  best  available  existing  commercial 
technology,  and  developing  new  technology  in  partnership  with  industry. 
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The  scope  of  A2XS  prototyping  projects  might  include  any  topic  generally  consistent  with  the 
RDT&E  of  (explain  scope  of  intended  prototype  projects,  erring  on  the  side  of  broader  rather 

than  restrictive) _ .  Generally  government  sponsors  with  use  RDT&E  appropriations  for  these 

activities.  However,  use  of  procurement  or  O&M  appropriations  might  be  appropriate  to  pay 
for  prototyping  projects  for  testing  and  evaluation  of  non-developed  items,  or  Commercial  Off- 
the-Shelf  (COTS)  items,  under  conditions  explained  in  the  Financial  Management  Regulations. 


USG  is  considering  applying  the  following  authorities  regarding  options  for  award,  which  are 
enumerated  in  10  USC  2371b: 

•  $250M  accumulative  funding  ceiling  may  be  apportioned  across  a  specified  period  of 
performance. 

•  Contractors  receiving  awards  under  A2XS  will  contribute  one  third  cost  share  unless  one 
of  the  following  applies 

o  Awardee  is  a  "small  business,"  "non-traditional  Defense  contractor,"  or  a  team 
consisting  entirely  of  small  businesses  and  non-traditional  Defense  contractors, 
as  defined  in  the  statue. 

o  Team  receiving  the  award,  which  may  include  "traditional  Defense  contractors" 
includes  at  least  one  non-traditional  Defense  contractor  who  will  contribute 
significantly  to  project  objectives 

•  USG  may  choose  to  make  direct  award  for  production,  without  further  competition, 
following  prototype  projects  that  successfully  achieve  defined  exit  criteria. 

Source  Selection: 

USG  seeks  comments  on,  or  alternatives  to,  the  following  notional  source  selection  criteria: 

•  Bidding  consortia  must  be  not-for-profit  organizations.  They  may  or  may  not  be 
formally  incorporated  as  such,  e.g.  per  under  26  USC  501(c)  3. 

•  Bidding  consortia  must  designate  an  individual  or  organization  to  serve  as  consortium 
manager,  or  specify  an  alternative  governance  structure  that  represents  the  best 
interest  of  the  members  at  large,  and  may  serve  as  an  independent  facilitator  of,  but  not 
participant  in,  competitions  for  prototyping  projects. 

•  Consortium  managers,  if  any,  must  specify  how  they  will  work  closely  with  the 
government  to  help  shape  and  broker  prototyping  projects,  and  represent  the  business 
interests  of  the  consortium  members  without  conflict  of  interest.  Consortium  mangers 
incorporated  as  not-for-profit  for  scientific  purposes  under  26  USC  501(c)  3  may  cite 
that  status  as  evidence.  However  not-for-profit  status  is  not  a  requirement. 

•  Bidding  consortia  should  exhibit  the  following  characteristics 
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o  Low  barrier  to  entry,  including  low  membership  fees  and  concise  membership 
agreement  that  is  intuitive  to  non-Defense  industry 
o  Rapid  membership  vetting  and  processing 
o  Members  provide  broad  coverage: 

■  Of  technical  topics  of  interest  to  XYZ  community 

■  Of  engineering,  analysis,  and  management  services 

■  Across  traditional,  non-traditional,  large,  and  small  firms 

•  Consortium  Managers  might  provide  evidence  of  ability  to  help  assist  government 
objectives  by,  e.g.: 

o  Performing  fiduciary  tasks  associated  with  transferring  funds  from  government 
to  members 

o  Performing  targeted  recruiting  of  new  members 
o  Facilitating  teaming  among  members 

o  Providing  services  associated  with  domain  technological  expertise  such  as  the 
following 

■  Performing  testing  and  other  forms  of  validation  and  verification 

■  Capturing  and  documenting  standards  and  best  practices 

■  Performing  cost  capability  analysis 

■  Performing  training 

■  Facilitating  technical  exchange 

o  Assisting  USG  and  members  to  leverage  acquisition  flexibility  allowed  under  10 
USC  2371b 

o  Suggesting  innovative  fee  structures  designed  to  incentives  achievement  of 
government  objectives,  e.g. 

■  Fixed  transaction  fee  (same  fee  regardless  of  price  of  project) 

■  Tiered  transaction  fees  (fees  vary  for  projects  that  fall  within  different 
price  ranges)  based  on  project  price,  and/or  services  rendered,  adjusted 
to  exclude  hardware,  material,  and  travel  costs2* 

■  Tiered  transaction  fees  based  on  total  project  price 

■  Percentage  of  price  up  to  a  maximum  $  amount  based  on  project  price 
adjusted  to  exclude  hardware,  materials,  and  travel  costs* 

■  Percentage  of  project  price  up  to  a  maximum  $  amount 


2  The  exclusion  of  hardware,  materials,  and  travel  from  the  fee  calculation  is  intended  to 
capture  labor  costs  as  the  main  indicator  of  risk  in  a  given  project,  which  may  lead  to  more 
challenging  management  during  performance  of  a  given  project.  Projects  that  consist  in  large 
part  of  material,  hardware,  and/or  travel  are  perceived  to  be  less  risky  and  require  less 
potential  for  substantial  involvement  on  the  part  of  the  Consortium  Manager. 
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The  Government  anticipates  responses  to  this  announcement  from  interested  parties,  eligible 
entities  or  groups  of  entities,  to  include:  industry,  academic,  non-profit,  and  not-for-profit 
organizations  for  research  and  development.  The  Government  especially  seeks  feedback  from 
organizations  that  may  be  interested  in  being  a  consortium  manager.  An  interested  parties  list 
will  be  created,  to  include  entities  and  groups  of  entities  that  would  be  considering  performing 
the  role  of  soliciting,  awarding,  and  managing  prototype  projects,  as  well  as  individual  entities 
that  are  interested  in  joining  the  resulting  consortium.  Responses  can  be  sent  to _ 

Responses  (email  encouraged)  should  at  a  minimum,  provide  the  following: 


1.  Name  of  consortium  and  consortium  manager  (if  any)  and  website  URLs. 

2.  Telephone  number  and  e-mail  address  for  each  POC,  CAGE  Code,  and  any  other 
pertinent  information 


3.  No  more  than _ (maybe  2500)  words  of  feedback  about  the  draft  USG 

objectives  and  recommended  processes  and  practices,  including  explanation  of  how  the 
objectives  would  be  addressed  and  outcomes  measured  and/or  recommended 
alternative  approaches. 


4.  In  no  more  than _ (maybe  1500)  words,  explain  corporate 

competencies  and  past  performance  with  respect  to  USG  objectives  and  source 
selection  criteria.  Explain  experience  with  regard  to  consortium  management  in  context 
with  government  and/or  commercial  applications. 

5.  Describe  experience  in  XYZ  programs  and/or  any  experience  with  XYZ  related  technologies 
and  processes  from  current  and  historical  sources  across  Government,  industry,  and  academia; 
experience  contributing  to  the  XYZ  domain;  and  experience  promoting  efficient  and  effective 
XYZ-related  information  sharing  and  collaboration. 

Please  submit  all  pages  as  a  single  (.doc  or  .pdf)  file.  Eliminate  or  minimize  any  proprietary 
information.  CLEARLY  MARK  all  proprietary  information.  Note  that  all  submissions  become 
Government  property  and  will  not  be  returned. 

The  USG  anticipates  an  ongoing  dialog  with  potential  bidders,  and  expects  that  dialog  to 
result  in  refinements  to  this  solicitation.  At  any  time  during  this  process  the  government  may 
choose  to  refer  to  the  current  version  of  this  solicitation,  and  announce  intentions  to  make 
awards.  If  and  when  that  occurs,  offerors  may  announce  that  their  prior  submissions  serve  as 
bids,  or  submit  supplements  or  new  proposals. 
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All  information  is  to  be  submitted  at  no  cost  or  obligation  to  the  Government.  The  U.S. 
Government  is  not  obligated  to  notify  respondents  of  the  results  of  this  announcement.  The 
U.S.  Government  reserves  the  right  to  reject,  in  whole  or  in  part,  any  private  sector  input,  as  a 
result  of  this  announcement.  If  a  formal  solicitation  is  generated  at  a  later  date,  a  separate 
solicitation  notice  will  be  published.  Interested  parties  are  responsible  for  adequately  marking 
proprietary  or  competition  sensitive  information  contained  in  their  response.  No  sensitive  or 
classified  information  will  be  discussed.  Foreign-owned,  controlled,  or  influenced  firms  are 
advised  that  security  restrictions  may  apply  that  may  preclude  their  participation  in  these 
efforts. 

Contracting  Office  Address: _ 

Primary  Point  of  Contact: _ 

Secondary  Point  of  Contact: _ 

Contract  Specialist: _ 

Notional  Statement  of  Objectives  (SOO)  for  Prototype  Project 
Transaction 


Please  consider  the  following  notional  example  of  a  SOO  for  a  prototype  project  as  a  potential  template. 


Statement  of  Objectives  for  XYZ  Enterprise  Open  System  (Phase  1)  Prototyping  Project 

Appendices: 

A.  References 

B.  ABC  Functional  Requirement  Specifications 

C.  Enterprise  Open  System  Requirement  Specifications  for  Cloud  Enabled  Software-as-a-Service  and 
Platform-as-a-Service. 

D.  Legacy  Data  Source  Description  and  Network  Topology 

E.  Government  Furnished  Equipment/Information 

Bottom  Line  Up  Front. 

The  XYZ  Program  Office  may  choose  to  execute  Adaptive  Acquisition  prototyping  project(s),  under 
"Other  Transactions"  Authority  per  US  Code  2371b,  as  a  means  to  incrementally  evolve  a  cloud-enabled, 
Enterprise  Open  System.  (See  Reference  A)  Successful  prototyping  projects  may  or  may  not  lead  to 
direct  award  of  transaction(s)  or  contract(s)  for  production.  The  first  phase  of  the  project  is  to  provision 
an  authorized  and  accredited,  cloud  enabled,  ABC  capability  (ABC  is  some  pilot  service  offered  via  the 
new  open  system).  Functional  requirements  for  the  ABC  capability  are  specified  at  Appendix  B.  Tasks 
for  this  XYZ  enterprise  open  system  prototyping  project  might  include: 

1.  Develop  a  government  reference  architecture  for  a  USG  cloud  enabled  Enterprise  Open  System. 
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2.  Provision  a  virtual  systems  integration  laboratory  (vSIL),  including  "plug  test"  capability,  that 
instantiates 

a.  An  early  "pluggable"  prototype  version  of  the  Government  Reference  Architecture 
(GRA)  Platform-as-a-Service  (PaaS)  and  Software-as-a-Service  (SaaS)  infrastructure  and 
middleware 

b.  ABC  SaaS  functionality. 

c.  Virtual  Open  Standard  Security  Services  (VOS3) 

3.  Prepare  all  Risk  Management  Framework  (RMF)  artifacts  required  to  achieve  Authorization  and 

Accreditation  (A&A)  to  operate  the  prototype  capability  on _ Network(s)  (" _ "  refers  to 

relevant  operational  and/or  test  networks) 

These  six  tasks  might  be  performed  as  either  individual  or  combined  tasks  by  one  or  more  performers, 
or  teams  of  performers. 

Enterprise  Open  System  Prototyping  Concept 

Today  the  US  Government  generally,  and  DoD  in  particular,  maintain  an  amalgam  of  purpose-built 
networked  information  systems.  (See  notional  depiction  at  figure:  1)  The  mission  of  the  XYZ  Program 

Office  is  to _ (explain  relevance  of  XYZ  mission  to  developing  the  target  system.)  The  standalone, 

redundant,  structure  of  the  legacy  architecture  sub  optimizes  opportunities  for  cross  functional 
operational  and  fiscal  efficiencies.  Accordingly,  the  XYZ  Program  Office  aims  to  apply  Adaptive 
Acquisition  and  Open  System  Acquisition  to  iteratively  and  continuously  evolve  a  cloud-enabled, 
interoperable,  enterprise  system  according  to  observed  best  practices  from  industry  and  government. 
(See  references  A- _ .) 

As  for  any  information  system,  the  key  to  evolving  effective  operational  interoperability  will  be 
constructing  optimal  abstraction  layers  necessary  to  allow  disparate  sub-process  to  efficiently  find  each 
other,  and  conduct  high  value  transactions.  (See  figures:  2-4)  The  key  to  evolving  effective  engineering 
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Functions 

X 

Y 

Z 


Hardwired  Data  Source 


Hardwired  Data  Source 


Legacy  "closed"  system. 
Characterized  by  relatively 
"hardwired"  system 
components  and  information 
product  delivery. 


Figure  4:  Notional  depiction  of  legacy  closed  system. 

interoperability  will  be  performing  optimal  functional  decomposition  and  adapting  the  most  applicable 
open  standards  to  allow  plug-and-play  of  off-the-shelf  components.  The  discipline  of  mapping  critical 
business  workflows  and  then  optimizing  them  with  appropriate  evolutionary  technology  choices  is  often 
called  "Product  Line  Architecture"  (PLA),  and  is  an  established  effective  practice  in  many  industries 
including,  e.g.,  automobile,  consumer  electronics,  and  finance. 


Chris  Gunderson/SAF/AQ  OTI/(o)703-693-4177/(m)831-224-5182/25July2017 


40 


First  incremental  delivery  of  "open  system"  via  "one-off"  adapter. 


Figure  5:  First  incremental  delivery  of  "open  system"  modular  components  via  "one-off"  adapter.  May  also  require  one-off 
cloud-ready  hardware  device(s)  to  provide  temporary  infrastructure. 

In  this  sense  "evolution"  requires  a  "brownfield  approach."  I.e. _ (" _ "  refers  to  the  name  of 

the  enterprise  of  interest,  e.g.  USAF)  will  continue  to  operate  the  legacy  system,  while  XYZ  Program 
Office  executes  a  series  of  acquisition  projects  that  will  each  contribute  new  "DNA"  to  the  to-be 
"greenfield"  open  system.  In  general,  the  evolutionary  approach  is  as  follows: 

•  Establish  a  target  PLA  aligned  with _ (name  of  enterprise)  mission  and  business  priorities. 

•  Identify  worthy  components  of  legacy  architecture  that  might  adapt  to  open  interfaces. 

•  Design  and  budget  for  iterative  development  of  modular  components  and  open  interfaces  per 
PLA. 

•  Compose  multiple  small  procurements  that  require  performer(s)  to  develop  the  requisite 
interfaces  for  "plugging  in"  new  capability. 

•  Development  defined  as  RDT&E  investment  in  prototyping  projects  that  deliver  interoperable 
COTS/GOTS. 

•  "Transition"  is  typically  defined  as  negotiating  license  for  improved,  pre-approved,  COTS/GOTS. 
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"Sustainment"  is  defined  in  terms  of  COTS/GOTS  licenses  that  include  provisions  for  lifecycle 
tech  refresh. 


Functions 

A 

B 

C 

u 

Functions 

i 

Mil 

Continuing  incremental 
deliveries  of  open 
system  modular 
components  gradually 
adds  both  new 
functionality,  and  re¬ 
hosts  legacy 
functionality. 


Data  as  a  Service 


Figure  6:  Continuing  gradual  evolution.  Some  stovepipe  functionality  continues  while  other  functionality  becomes  available  in 
the  cloud.  Note  migration  away  from  legacy  data  sources  toward  standard  data  services. 


One  of  the  critical  enterprise  functions  is  ABC  capability. 


(Describe  ABC  capability  in 
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Legacy  system  retired.  Open  infrastructure,  is  mature. 


Figure  7:  USG  enterprise  product  line  architecture  is  mature.  Enterprise  is  fully  cloud  enabled.  Open  system  is  characterized  by 
continuous,  cost-effective,  improvement  and  broad  interoperability. 


context  with  current  state  of  the  art  and  policy  regime.)  (See  references _ - _ ) 


Today  the _  (name  of  legacy  enterprise  capability)  is  an  amalgam  of  standalone  systems,  which 

themselves  are  a  subset  of  a  larger  group  of  systems  that  address  requirements  such  as _ , _ 

and _ .  (" _ "  describe  functions  performed  by  relevant  legacy  stovepipe  systems.) 


The  legacy  ABC  capability _ (describe  status  quo  and  relevant  background  motivating 

the  prototype  project  to  instantiate  ABC  SaaS.) 


Consistent  with  the  evolutionary  process  described  above,  XYZ  Program  Office  aims  to  field  and  host 
ABC  SaaS  as  the  first  evolutionary  component  of  the  to-be  enterprise  open  system. 


Cyber  Security  as  a  Service 

The  USG's  and  DoD's  status  quo  approach  to  security,  including  especially  Authorization  and 
Accreditation  (A&A),  will  simply  not  support  the  targeted  open  system  efficiencies.  Each  rapidly 
developed  capability  would  be  delayed  at  least  many  months  and  its  cost  increased  by  at  least  hundreds 
of  thousands  of  dollars,  in  order  to  accomplish  A&A. 
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Status  quo  security  paradigms  are  based  on  physical  separation.  This  clashes  with  modern  software 
engineering  paradigms,  such  as  cloud,  that  are  based  on  logical  separation  through  virtual  technology. 

Accordingly,  the  proposed  open  system  approach  to  implementing  cyber  security,  i.e.  Virtual  Open 
Standard  Security  Services  (VOS3)  depends  on 

o  Implementing  open  standard  virtual  security  services  based  on  logical  separation  and  dynamic 
"need-to-share"  policies. 

o  Introducing  new  A&A  assurance  arguments  that  are  based  on  logical  separation/virtual 
technology/need-to-share  policies. 

The  Air  Force  Research  Lab,  Information  Directorate,  SecureView  cross-domain  "access  solution"  is  a 
rare  example  of  an  accredited  Cross  Domain  Solution  that  is  based  on  logical  separation  implemented 
with  virtual  technology.  Further,  members  of  the  government-sponsored  aviation  focused  standards 
bodies,  Unmanned  System  Control  Segment  Architecture  (UCS),  and  Future  Airborne  Capability  (FACE) 
are  developing  instantiations  of  the  same  approach.  These  efforts  may  provide  basis  for  the  requisite 
GRA  cyber  security  layer.  See  references  (l-K) 

Prototyping  Project  Objectives 

General  descriptions  of  XYX  Enterprise  Open  System  (Phase  1)  prototyping  tasks  follow: 


Figure  8:  First  incremental  delivery  of  USG  Business  Enterprise  Open  System  will  provision  ABC  functionality  via  scalable,  cloud 
enabled,  accredited  and  authorized  prototype  stack. 
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ABC  functionality 
delivered  on  first  instance 
of  scalable,  cloud 
enabled,  prototype  stack 


ABC  Functionality 


Overall  XYZ  Enterprise  Open  System  Phase  1  Objectives 

1.1.  Establish  commercial  market  incentivized  to  continuously  improve  the  Enterprise  Open  System 
(EOS) 

1.2.  Architect,  accredit,  authorize,  and  implement  ABC  SaaS  as  first  prototype  instance  of  USG  cloud 
enabled  EOS  architecture.  (See  notional  depiction  figure:  6.A) 

1.3.  Develop  roadmap  for  retiring  the  legacy _ systems. 

1.4.  Demonstrate: 

1.4.1.  Customization/configuration  options  at  all  layers  of  the  technical  stack. 

1.4. 2.  Pluggable  flexibility  to  reuse,  enhance,  and  expand  functionality  and  performance. 

1.4.3. Interoperability  with  legacy  and  non-legacy  systems. 
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1.5.  Develop  engineering,  programmatic,  and  business  models  to  enable  and  sustain  all  the  above. 


Prototyping  Procurement  Objectives 

2.1.  Expose  USG  requirements  to,  and  lower  barriers  for,  broad  community  of  traditional  and  non- 
traditional  technology  providers. 

2.2.  Accelerate  solicitation  to  award  timeline. 

2.3.  Perform  Analysis  of  Alternatives  (AoA),  trades  analysis,  source  selection,  and  performance 
monitoring  on  the  basis  of  demonstrated,  open  system  interoperability  and  functionality. 

2.4.  Develop  Intellectual  Property  Regimes  (IPR)  and  Data  Rights  that  incentivize  mutually  beneficial 
government-industry  partnership. 

2.5.  Transition  successfully  tested  prototype  applications  as  vendor-supported  off-the-shelf 
products. 

2.6.  Establish  growing  catalog  of  pre-approved,  off-the-shelf,  XYZ  EOS  products  and  services 


Measures: 

1.  Numbers  of  traditional  and  non-traditional  vendors  who  participate  and/or  compete. 

2.  Solicitation-to-Award  timeline.  Target:  30  days. 

3.  Yes/No  vendor  COTS  offering  is  available  via  convenient  procurement  vehicle  following 
successful  prototype  PlugTest. 

4.  Yes/No  agreed  Data  Rights  captured  in  standard  procurement  language. 

5.  Yes/No  USG  receives  enterprise  license  to  COTS  product  in  return  for  investment  in 
prototype  development. 

6.  Infrastructure  maintenance  cost  avoided,  and  reinvested  in  business  process  improvement, 

by  consolidating  and  modernizing  IMCS  functionality.  (Target:  $ _ M  from _ to _ .) 

Reference: 

F.  Open  System  Acquisition  Practical  Guide. 

Q.  Adaptive  Acquisition  Practical  Guide 

Prototyping  Management  Objectives 

3.1.  Provision  continuous  feedback  loop  with  operational  customers. 

3.2.  Conduct  AoA,  demonstration,  development,  Test  &  Evaluation  (T&E),  Validation  &  Verification 
(V&V),  and  certification  in  parallel. 

3.3.  Minimize  risk  of  technology  perishability  by  executing  rapid,  iterative,  evolutionary, 
demonstration-to-delivery  cycles. 

Measures: 
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1.  Yes/no,  specified  operational  customer(s)  validates  prototype  test  results. 

2.  Yes/no,  project  execution  plan  documents  independent  schedules  and  budgets  for 
specified  parallel  activities  and  integration  events  and  accounts  for  dependencies. 

3.  Timeline  from  prototype  initial  demo  to  successful  prototyping  exit  criteria.  Target:  6 
months. 

4.  Timeline  from  prototype  exit  criteria  to  pre-approved  COTS.  Target:  6  months. 

5.  Yes/no,  acquisition  risk  management  artifacts  address  speed-to-capability  as  critical  risk 
factor,  and  explain  credible  mitigation  factors. 

Reference: 

H.  Open  System  Acquisition  (OSA)  Project  Execution  Plan  (PEP)  Draft  Data  Item  Description 
(DID):  SAF  AQ  OSA-DID-PEP-3-20-15. 

Prototyping  Engineering  Objectives 

4.1.  Evolve  cloud  enabled  EOS  Government  Reference  Architecture  (GRA).  I.e.  develop  and 
document  technical  specifications,  including  Interface  Control  Document  (ICD),  for  product  line 
architecture  optimized  to  assure  "pluggable"  component  interoperability  aligned  with  specific 
mission  and  business  objectives  consistent  with  reference  documents. 

4.2.  Establish  persistent,  open,  cloud-enabled,  virtual  system  integration  laboratory  (vSIL)  that: 

4. 2.1.  Allows  dynamic  provisioning  of  capability  within  virtual  machines 

4. 2. 2.  Dynamically  partitions  accessibility  across  virtual  domain  based  on  specified  security  and 
privacy  policies 

4. 2. 3.  Hosts  design  time,  build  time,  and  run  time  government-provided  reference  instances  of 
target  technical  architectures  -  i.e.  instantiates  government  reference  PLA. 

4. 2.4.  Hosts  PlugTest  services.  Includes,  but  not  limited  to: 

4. 2.4.1.  Live,  Virtual  and  Constructive  (LVC)  models  and  simulations. 

4. 2.4.2.  Conformance  test  kits 

4. 2.4.3.  Online  library  of  standard  test  tools 

4. 2.4.4.  Cloud  provisioning  dashboard(s) 

4. 2.4.5.  Test  harness  based  on  reference  implementation(s) 

4. 2. 4. 6.  Test  data  sets 

4. 2. 5.  Provision  runtime  instance  of  GRA  within  PlugTest  harness. 

4. 2. 6.  Provision  runtime  instances  of  item  management  control  functionality  within  PlugTest 
harness. 

4.3.  Establish  cyber  security  capability,  authority,  and  documentation  such  that  successful  XYZ  EOS 
PlugTests  are  tantamount  to  cyber  security  authorization  and  accreditation  for  the  artifact 
under  test. 
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4.4.  Establish  an  XYZ  EOS  collaborative  engineering  forum,  composed  of  experts  from  government 
agencies,  and  traditional  and  non-traditional  companies,  qualified  and  incentivized  to  advise 
development  of  XYZ  EOS  architecture  and  standards. 

4.5.  Instantiate  XYZ  EOS  GRE  generally,  and  ABC  functionality  specifically,  into  operational 
prototype  reference  implementation. 


vSIL  Plug  Test  Harness 


Software  as  a  Service 


Open  Std  Middleware 


Artifact  Under  Test 


Platform  as  a  Service 


Figure  9:  The  virtual  Systems  Integration  Lab  includes  an  online,  pluggable,  instance  of  the  cloud  enabled  Government  Reference 
Architecture  connected  to  test  services. 


Measures: 

1.  Yes/No  consortium  engineering  activity  results  in  measurable  evolution  of  GRA  and 
associated  standards. 

2.  Yes/No,  ICDs  for  GRA,  PlugTest  harness,  and  ABC  SaaS,  exist  and  specify  functional  modules 
and  open  standard  interfaces  aligned  with  measurable  operational  and  system  performance 
objectives. 

3.  Yes/No,  design-time,  build-time,  and  runtime  instances  of  GRA,  ABC  SaaS,  and  PlugTest 
harness,  exist  and  have  been  validated  and  verified  as  follows: 
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a.  No  less  than _ %  (e.g.  80%)  of  lifecycle  costs  of  deployed  and  planned  systems  and 

components  shall  be  for  purchase  of  COTS/GOTS  items  and/or  services. 

b.  Platform-as-a-service  and  software-as-a-service  performance  for  XYZ  EOS  generally,  and 
ABC  SaaS  in  particular,  shall  comply  with  operational  and  system  performance  standards 
for  operational  availability,  scalability,  security,  interoperability,  latency,  as  specified  in 
Appendix  C. 

c.  ABC  SaaS  functionality  shall  comply  with  requirements  specified  in  Appendix  B. 

d.  A  trained  operational  user  shall  locally  update  SaaS  capability  feature  within _ minutes. 

The  system  shall  provision  workflow  required  to  validate  and  implement  a  global  update 
within _ day(s). 

e.  A  third  party  developer  shall  install  a  new  application  within  PlugTest  versions  of  XYZ  EOS 
GRA  within  specified  time  window.  (E.g.  Target:  1  week.) 

Reference: 

I.  Open  System  Acquisition  (OSA)  Product  Line  Architecture  (PLA)  Draft  Data  Item  Description 

(DID):  SAF  AQ  OSA-DID-PLA-3-20-15 

J.  Open  System  Acquisition  (OSA)  Plug  Test  Plan  (PTP)  Draft  Data  Item  Description  (DID):  SAF  AQ 

OSA-DID-PTP-3-20-15. 

G.  NIST  SP  500-291,  Cloud  Computing  Standards  Roadmap,  chapters  6  &  7. 

Logistics  Objectives 

5.1.  Reduce  Total  Cost  of  Ownership  per  Capability 

5.2.  Conduct  continuous  technology  refresh. 

5.3.  Provision  for  user  and  maintenance  training. 

Measures: 

1.  Yes/No,  vendor-supported  COTS  applications  are  available  on  convenient  procurement 
schedule,  e.g.  GSA,  at  known  cost. 

2.  Number  of  vendors  who  deliver  competing  capabilities. 

3.  Yes/No,  all  delivered  capabilities  have  robust  user  and  maintenance  documentation  and 
instructions. 

Reference: 

K.  Open  System  Acquisition  (OSA)  Information  Technology  Users  Guide  (ITUG)  Draft  Data  Item 

Description  (DID):  SAF  AQ  OSA-DID-ITUG-3-20-15. 
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Potential  XYZ  EOS  Phase  1  Prototyping  Tasks 

Tl.  Develop  Government  Reference  Architecture  for  a  XYZ  cloud  enabled  Enterprise  Open 
System. 

Perform: 

•  Collaborative  engineering  with  other  prototyping  project  participants 

•  Continuous  feedback  loop  with  operational  alpha  and  beta  community 

•  Market  analysis 

•  Model  Based  Systems  Engineering: 

o  Design  &  documentation 
o  Validation  and  Verification 

Deliver:  Product  Line  Architecture  Interface  Control  Documentation  and  other  specifications  per  draft 
Data  Item  Description  (DID)  SAF  AQOSA-DID-PLA-3-20-15.  Shall  addresses  broad  implementation  and 
tech  refresh  of  XYZ  GRA.  PLA  will  be  sufficiently  robust  to  support  implementation  of  core  technology 
into  various  form  factors  such  as  enterprise  cloud  services,  and  hand  held  mobile  devices. 

T2.  Provision  a  virtual  systems  integration  laboratory,  including  "plug  test"  capability,  that 
instantiates  an  alpha  version  of  the  GRA. 

Perform: 

•  Collaborative  engineering  with  other  prototyping  project  participants 

•  Continuous  feedback  loop  with  operational  alpha  and  beta  community 

•  Analysis  of  Alternative  cloud  hosting  platforms 

•  Development  and  installation  of, 

o  Portable  virtualized  PaaS/SaaS  middleware 
o  Test  cases  and  test  services 

•  Test  operations 
Deliver 

•  Prototype  software 

•  Software  Test  Report  per  draft  DID  SAF  AQ  OSA-DID-PTP-3-20-15 

•  Executable  software  application,  license,  user  manual  and  Concept  of  Operations  (CONOPS)  for 
specified  USG  applications  specification  per  draft  DID  SAF  AQ  OSA-DID-ITUG-3-20-15 

T3.  Develop  a  pluggable  prototype  instance  of  the  reference  architecture  that  provisions: 


T3  (a)  Generic  cloud-ready  PaaS  and  SaaS  functionality 

Perform: 

•  Collaborative  engineering  with  other  prototyping  project  participants 

•  Continuous  feedback  loop  with  operational  alpha  and  beta  community 

•  Fabrication  of  functional  prototype 

•  Validation  and  Verification  of  required  functionality  and  performance 
Deliver 

•  Prototype  hardware 

•  Software  Test  Report  per  draft  DID  SAF-AQ-BTCC-PT  -3-18-15 

•  Executable  software,  license,  user  manual  and  Concept  of  Operations  (CONOPS)  for  specified 
USG  applications  per  draft  SAF  AQ  OSA-DID-ITUG-3-20-15 
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T3  (b)  Specific  ABC  SaaS  functionality 

Perform: 

•  Collaborative  engineering  with  other  prototyping  project  participants 

•  Continuous  feedback  loop  with  operational  alpha  and  beta  community 

•  Development  of  item  management  software  application 

•  V&V  of  required  functionality  and  performance 
Deliver 

•  Software  Test  Report  per  draft  DID  SAF  AQ  OSA-DID-PTP-3-20-15 

•  Executable  software  application  with  verified  functionality  as  demonstrated  by  Item  Master 
under  CTMA,  with  license,  user  manual  and  Concept  of  Operations  (CONOPS)  for  specified  USG 
applications  per  draft  DID  SAF  AQ  OSA-DID-ITUG-3-20-15 

T3  (c)  Cyber  Security-as-a-Service. 

Perform: 

•  Collaborative  engineering  with  other  prototyping  project  participants 

•  Continuous  feedback  loop  with  operational  alpha  and  beta  community 

•  Development  of  Cyber  Security  as  a  Service  software 

•  V&V  of  required  functionality  and  performance 
Deliver 

•  Software  Test  Report  per  draft  DID  SAF  AQ  OSA-DID-PTP-3-20-15 

•  Executable  software  application,  license,  user  manual  and  Concept  of  Operations  (CONOPS)  for 
specified  USG  applications  per  draft  DID  SAF  AQ  OSA-DID-ITUG-3-20-15 

T4.  Prepare  all  Risk  Management  Framework  artifacts  required  to  achieve  authorization  and 
accreditation  to  operate  the  prototype  capability  on _ network(s). 

Perform: 

•  Collaborative  engineering  with  other  prototyping  project  participants 

•  Development  of  assurance  arguments  and  documentation  for  A&A  XYZ  EOS  GRA  based  on 
logical  separation. 

Deliver 

•  Accreditation  and  Authorization  plan  that  achieves  interim  authority  to  operate  within  9  months 
of  task  start,  and  authority  to  operate  within  twenty-four  months  of  task  start 

•  RMF  A&A  artifacts  presented  as  required 


Conceptual  Prototyping  Project  Execution  Plan  (Budget  and  Schedule  are  Notional) 
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Task  Name 


Cost 


-  XYZ  Enterprise  Open  System  Phase  1 

$2,100,000.00 

-  Compose  Project  Team 

$0.00 

Identify  PM 

so.oo 

Identify  Chief  Architect 

$0.00 

Identify  transaction  vehicle  and 
Agreements  Officer 

50.00 

Identify  operational  alpha/beta 
community 

SO.OO 

Identify  Accreditation  Official 

50.00 

-  Establish  Consortia  of  Commercial 
BEOS  Stakeholders 

$0.00 

Itentify  4  invite  likely  members 

so.oo 

Host  information  exchage  calls 

so.oo 

-  Solicitation 

$0.00 

Publish  draft  SOO  for  comment, 
refine  and  iterate 

so.oo 

Request  and  review 
whitepapers 

so.oo 

Issue  RFPs 

so.oo 

-  Develop  XYZ  EOS  Product  Line 
Architecture  (PLA) 

$400,000.00 

Award  transaction 

$400,000.00 

*  Alpha/Beta  feedbacks  project  ci 

$0.00 

Market  Analysis 

so.oo 

-  MBSE 

$0.00 

Preliminary  Design  ver  0.0 

so.oo 

Preliminary  V&V 

so.oo 

Final  Design  ver  1.0 

so.oo 

Final  VSV 

so.oo 

Qtr  3,2017 

Qtr  4, 2017 

Qtr  1. 2018 

Qtr  2, 2018 

Jul  Aug  Sep 

Oct  Nov  |  Dec 

Jan  j  Feb  Mar 

Apr  May  |  Jun 

Qn  2018 
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o  isi  o  a  o  h  m  ta  o  tsi 


0 


Task  Name 

Cost 

-  Develop  EOS  vSIL 

$600,000.00 

Award  transaction 

S600, 000.00 

+  Alpha'Beta  feedback  &  project  ci 

$0.00 

AoA  of  cloud  hosting  options 

50.00 

Select,  develop,  and  instal  prototypel 
PaaS  &  SaaS  Middleware  ver  0.0 

SO.OO 

Select,  develop,  and  install  prototype 
test  services  ver  0.0 

so.oo 

Preliminary  V&V 

SO.OO 

Select,  develop,  and  instal  prototypel 
PaaS  &  SaaS  Middleware  ver  1.0 

so.oo 

Select,  develop,  and  install  prototype 
test  services  ver  1.0 

so.oo 

Final  V&V 

so.oo 

-  Develop  ABC  SaaS 

$100,000.00 

Award  transaction 

5100,000.00 

+  Alpha'Beta  feedback  &  project  ci 

so.oo 

Select,  develop,  and  install  ABC 

SaaS  s/w  ver  0.0 

so.oo 

Preliminary  V&V 

so.oo 

Select,  develop,  and  install  ABC 

SaaS  s/w  ver  1.0 

so.oo 

Final  V&V 

so.oo 

-  Develop  virtual  cyber  security 
services 

$500,000.00 

Award  transaction 

5500,000.00 

+  Alpha/Beta  feedback  &  project  ci 

so.oo 

Select,  develop,  and  Install  prototype 
cyber  security  service!  s/w  ver  0.0 

so.oo 

Preliminary  V&V 

so.oo 

Select,  develop,  and  install  prototype 
cyber  security  services  s/w  ver 

1.0 

so.oo 

Final  V&V 

0.00 

-  Develop  and  present  RMF  artifacts 

$500,000.00 

Award  transaction 

S500, 000.00 

+  Alpha 'Beta  feedback  &  project  ci 

so.oo 

Catagorize  system  (initiate  Security 

Plan  (SP)) 

so.oo 

Select  controls 

so.oo 

Prepare  Security  Assessment 

Report  (SAR) 

so.oo 

Submit  Secuirty  Authorization 

Package  (SAP) 

so.oo 

Process  Interim  Authority  to  Operate 

so.oo 

Process  Authority  to  Operate 

so.oo 
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APPENDIX  A:  References 


A.  Adaptive  Acquisition  Practical  Guide,  Gunderson,  Air  Force  Institute  of  Technology 

B.  Open  System  Acquisition  Practical  Guide,  Gunderson,  Naval  Postgraduate  School 

C.  NIST  SP  500-291,  current  version,  Cloud  Computing  Standards  Roadmap,  Chapters  6&7 

D.  Open  System  Acquisition  (OSA)  Project  Execution  Plan  (PEP)  Draft  Data  Item  Description 
(DID):  SAF  AQ  OSA-DID-PEP-3-20-15. 

E.  Open  System  Acquisition  (OSA)  Product  Line  Architecture  (PLA)  Draft  Data  Item  Description 
(DID):  SAF  AQ  OSA-DID-PLA-3-20-15 

F.  OSA  Plug  Test  Plan  (PTP)  Draft  Data  Item  Description  (DID):  SAF  AQ  OSA-DID-PTP-3-20-15. 

G.  OSA  Information  Technology  User's  Guide  (ITUG)  draft  DID  SAF-AQ-DID-ITUG  -3-18-15 

H.  DOD  Instruction  8510.01,  Risk  Management  Framework 

I.  Cloud  Friendly  Virtual  Information  Assurance  Security  and  Cross  Domain  Assurance, 
Gunderson  Naval  Postgraduate  School 

J.  UAS  Control  Segment  Working  Group  Subgroup  6  Open  System  Environment  for  Safety  and 
Information  Assurance:  Application  Platform  Requirements 

K.  SecureView:  Government/Industry  Collaboration  Delivers  Improved  Levels  of  Security, 
Performance,  and  Cost  Savings  for  Mission-Critical  Applications,  Durante  &  Wood,  AFRL 
Information  Directorate 

L.  Program-specific  documentation 

M.  ... 
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APPENDIX  B:  ABC  SaaS  Functional  Requirement  Specification 


USG  must  do  a  careful  job  of  specifying  objective,  testable,  measures  of  effectiveness  in  terms  of 
mission  and  business  outcomes,  and  tightly  correlated  system  and  process  level  measures  of 
performance. 
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APPENDIX  C:  XYZ  Enterprise  Open  System  requirement  mission  and  system  performance 
specifications  for  SaaS  and  PaaS. 


Recommend  get  input  from  the  engineering  forum  created  by  the  solicitation  on  what  these  should  look 
like: 


1.  Operational  Availability 

2.  Scalability 

3.  Workflow  latency 

4.  Engineering  interoperability  (how  quickly  and  conveniently  can  new  components  be  configured 
and  retired?) 

5.  Operational  interoperability  (can  all  appropriate  nodes  participate  in  required  workflows 
without  impacting  overall  system  efficiency?) 

6.  Security  (RMF  controls) 

7.  Lifecycle  cost  (e.g.  annual  lifecycle  cost  of  all  system  components  shall  be  known  prior  to 
deployment) 

8.  Lifecycle  tech  refresh  frequency/method,  e.g.  software  licenses  shall  include  provision  for 
continuous  tech  refresh. 
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APPENDIX  D:  Legacy  Data  Source  Description  and  Network  Topology 


Need  to  provide  detailed  descriptions  of  legacy  data  types,  associated  interfaces,  and  network  access 
information. 
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APPENDIX  E:  Government  Furnished  Equipment/Information 


1.  E.g.  license  for  legacy  ABC  software 

2.  E.g.  access  to  cloud  hosted  development  environment 

3.  Other? 
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Offeror  Response  Format 


The  U.S.  Government  (USG)  may  choose  to  award  funds  currently  estimated  at _ ,  across  a  period  of 

_ ,  to  develop  and  field  solution(s)  to  satisfy  the  accompanying  Statement  of  Objectives  (SOO). 

Toward  that  end,  the  USG  intends  to  perform  a  "rolling  solicitation"  to  support  market  analysis,  analysis 
of  alternatives,  trades,  and  (potentially)  source  selection.  That  is,  USG  seeks  robust,  continuing, 
communication  with  potential  awardees  in  order  to  refine  the  SOO  based  on  discovered  state  of  the 
market,  and  advice  from  industry.  When  and  if  the  USG  achieves  a  threshold  comfort  level,  it  will 
announce  intention  to  conduct  source  selection  and  make  an  award. 

Questions  1-6  are  relevant  to  market  analysis,  analysis  of  alternatives,  and  source  selection.  Questions 
7-10  are  relevant  for  source  selection  when  and  if  the  USG  announces  intent  to  award.  USG  expects 
that  offerors  will  provide  essential  elements  of  information  as  objectively,  and  succinctly  as  possible, 
without  embellishment.  Responses  that  exceed  word  limits  may  be  considered  non-responsive.  You 
may,  but  are  not  required  or  expected  to,  provide  hyperlinks  to  more  detailed  information.  You  are 
encouraged  to  ask  questions  and  propose  modifications  to  government  concepts  and  approaches. 

USG  may  or  may  not  follow  up  with  request  to  discuss,  or  for  more  information. 

USG  will  make  any  information  provided  to  one  respondent  available  to  all  respondents. 


1.  Please  identify  each  firm  participating  in  your  current  response  by  name,  URL,  and  point  of  contact. 
Indicate  which,  if  any,  team  member  will  serve  as  prime.  (USG  recognizes  that  respondent  teams  may 
evolve  as  the  solicitation  evolves.) 


2.  Please  describe  your  proposed  approach  in  500  words  or  less.  Note  that  USG  has  strong  preference 
to  compose  capability  with  Off  the  Shelf  (OTS)  building  blocks.  (See  note  #3  in  Appendix  A  for  definition 
of  OTS.)  Hence,  to  the  extent  it  is  relevant,  summarize  your  approach  in  in  terms  of  how  you  would: 

a)  Identify  baseline  OTS  capability 

b)  Identify  gap  between  existing  OTS  and  total  requirement 

c)  Rapidly  compose  and  deploy  OTS-based  aspects  of  solution 

d)  Invent  new  capability,  transition  it  as  OTS,  and  connect  it  to  OTS-baseline 

e)  Conduct  continuous  lifecycle  tech  refresh 

Please  include  and  highlight  proposed  changes  to  USG  concept,  objectives,  and  plans. 
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3.  If  your  approach  includes  use  of,  or  modifications  to,  your  own  existing  OTS  products  and  services  (as 
defined  in  Appendix  A:  note  #3),  please  identify  products  and  services  by  name  and  hyperlink  reference 
to  ordering  procedures,  license,  feature  descriptions,  lifecycle  cost,  tech  refresh  process,  etc. 


4.  Please  specify  any  existing  government  procurement  vehicles  through  which  your  OTS  offering(s) 
described  above  may  be  obtained. 


5.  Please  specify  any  current  and  past  cyber  security  or  flight  worthiness  authorizations,  certifications,  or 
accreditations  associated  with  your  OTS  offering(s)  identified  above. 


6.  Please  explain,  in  250  words  or  less,  how  you  think  the  USG  should  validate,  verify,  and  certify  your 
solution  for  performance  against  the  USG's  stated  objectives. 


Answers  to  following  questions  are  mandatory  only  when/if  USG  announces  intent  to  make  an  award, 
and  you  intend  to  make  an  offer.  You  are  welcome  to  submit  this  information  at  any  time  during  the 
rolling  solicitation.  If  you  do  so  prior  to  USG  announcement  of  intent  to  award,  you  will  be  asked  to 
make  a  statement  verifying  that  your  earlier  submission  remains  valid. 

7.  If  member(s)  of  your  bidding  team  qualify  as  non-traditional  Defense  contractors  (per  Appendix  A: 
note  #1),  please  specify  which  companies.  Please  also  explain  how  contributions  by  non-traditional 
team  members  are  "significant"  in  100  words  or  less. 


8.  If  neither  you,  nor  any  significantly  contributing  bidding  team  members  qualify  as  nontraditional 
defense  contractors,  are  you  willing  to  contribute  one  third  of  the  project  costs  under  terms  of  10  USC 
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2371b  (see  Appendix  A:  note  #2)?  If  so,  please  summarize  your  proposed  1/3  contribution  in  100  words 
or  less,  e.g.  time  and  material,  in  kind  contribution  (specify  contribution  and  value),  etc. 


9.  Does  your  bid  comply  with  all  security  clearance  requirements  specified  in  the  solicitation? 


10.  If  members  of  your  bidding  team  have  performed  successfully  on  recent,  relevant  projects 
(government  sponsored  or  otherwise)  please  concisely  describe  performance  on  no  more  than  three  of 
those  projects.  For  each  project  description,  in  200  words  or  less,  include: 

a)  Performing  company(ies)  name(s) 

b)  Project  name  and  period  of  performance 

c)  Name  and  contact  information  for  a  sponsoring  official  who  can  verify  your  contribution 

d)  Performing  company(ies)  role  in  the  project 

e)  Measures  of  success  as  defined  in  Appendix  A:  note  #4,  or  other  relevant,  objective  measures 
of  success 

f)  Relevance. 

g)  Hyperlinks  to  more  detail  (optional.) 

Regarding  measures  of  success,  different  metrics  apply  to  different  project  objectives  and  different 
performer  tasks.  USG  expects  that  it  is  unlikely  that  any  one  vendor  would  have  achieved  more  than  a 
small  number  of  the  specified  measures  of  success  on  any  particular  project.  Validated  relevance  of 
performance,  with  quality  of  performance  verified  as  objectively  as  possible,  is  more  important  than 
quantity.  Lack  of  prior  performance  does  not  disqualify  a  bid. 

Notional  Example 

l.a.  123  Company  Inc. 

l.b.  ABC  Financial  Trading  System  upgrade  circa  2014-16. 

l.c.  Mary  Noesme,  555-555-5555,  m.n@ XYZ.com. 

l.d.  Prime  contractor,  and  stock  trading  application  s/w  developer. 

l.e.  System  achieved  targeted  objective  of  10%  reduction  in  time-to-decision;  as 

prime,  123  Company  was  awarded  bonus  for  early  system  delivery;  software 

apps  for  financial  trades  has  half-life  of  about  1  year,  and  123  company 

delivered  developed  software  as  sustained,  certified,  cloud-enabled,  operational 

OTS  capability  within  8  months  of  contract  award,  with  established  process  for 

updating  software  no  less  than  annually. 
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l.f.  Although  ABC  Financial  Trading  System  is  a  commercial  application,  it  is 
essentially  a  highly  secure,  low  latency,  multi-node,  multi-level,  command  and 
control  system  aimed  at  informing  and  expediting  critical  tactical  decisions, 
l.g.  See:  www.l23/abcwhitepaper.com 
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APPENDIX  A:  Explanatory  Notes 


1.  Definition  of  "nontraditional  defense  contractor."  US  Code  Title  10  Section  2302(9)  defines  a 
"nontraditional  defense  contractor"  as:  "...  an  entity  that  is  not  currently  performing  and  has  not 
performed,  for  at  least  the  one-year  period  preceding  the  solicitation  of  sources  by  the  Department  of 
Defense  for  the  procurement  or  transaction,  any  contract  or  subcontract  for  the  Department  of  Defense 
that  is  subject  to  full  coverage  under  the  cost  accounting  standards  prescribed  pursuant  to  section  1502 
of  title  41  and  the  regulations  implementing  such  section..." 

2.  Cost  share  requirements  per  10  USC  2371b.  “...  no  official  of  an  agency  enters  into  a  transaction  ... 
unless  one  of  the  following  conditions  is  met: 

(A)  There  is  at  least  one  nontraditional  defense  contractor  participating  to  a  significant  extent  in 
the  prototype  project. 

(B)  All  significant  participants  in  the  transaction  other  than  the  Federal  Government  are  small 
businesses  or  nontraditional  defense  contractors. 

(C)  At  least  one  third  of  the  total  cost  of  the  prototype  project  is  to  be  paid  out  of  funds  provided 
by  parties  to  the  transaction  other  than  the  Federal  Government. 

(D)  The  senior  procurement  executive  for  the  agency  determines  in  writing  that  exceptional 
circumstances  justify  the  use  of  a  transaction  that  provides  for  innovative  business  arrangements  or 
structures  that  would  not  be  feasible  or  appropriate  under  a  contract,  or  would  provide  an 
opportunity  to  expand  the  defense  supply  base  in  a  manner  that  would  not  be  practical  or  feasible 
under  a  contract..." 

3.  Definition  of  "Off  the  Shelf."  "Off  the  Shelf"  (OTS)  capability  is  defined  as  an  existing  product  or 
service  that  has  all  of  the  following  characteristics: 

a.  Is  available  as  a  standard  catalog  item. 

b.  Can  be  obtained  through  an  existing  convenient  procurement  vehicle. 

c.  Has  a  published,  verified,  lifecycle  cost. 

d.  Has  a  published  and  verified  lifecycle  tech  refresh  process. 

e.  Configures  in  the  target  environment  in  a  specified  short  period  of  time. 

4.  Measures  of  a  successful  government  adaptive  acquisition.  A  "successful"  adaptive  acquisition  is 
defined  as  an  appropriately  funded,  well  managed,  engineering  project  that  delivers  useful  lifecycle 
supported,  up-to-date  technology,  cost  effectively,  to  a  satisfied  operational  customer,  in  time  to  make 
a  difference.  Successful  adaptive  acquisitions  may  be  categorized  by  measures  such  as,  but  not  limited 
to,  the  following: 
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a.  Mission  level  Measures  of  Effectiveness  (MOE).  Requirement  is  to  satisfy  threshold  values 
of  improvements  over  baseline  values  of  specified  MOE.  Examples  of  MOE  include:  Probability  of 
Detection,  Probability  of  Kill,  Operational  Availability,  Speed-to-Decision,  Cycle-Time-Compression,  and 
Accident  Rate. 

b.  Acquisition  process  level  Measures  of  Performance  (MOP).  Requirement  is  to  satisfy 
threshold  levels  of  adaptive  acquisition  process  level  MOP  such  as  the  following: 

(1)  Systems  Engineering  Plan,  which  addresses  scheduling,  risk-reward  management, 
development,  test,  and  certification,  delivers  technology  within  its  perishability  half-life,  and 
project  achieves  schedule.  Measures  are  verified  estimate  of  technology  half-life,  plus 
scheduled  vs.  actual  award-to-delivery  time. 

(2)  Procurement  plan  includes:  a)  relatively  small  RDT&E  budget  and  incentivized 
procurement  vehicle(s)  to  encourage  enhancement  of  OTS  products  and  services  to  meet 
government  requirements;  b)  relatively  large  O&M  budget  and  fixed  price  procurement 
vehicle(s)  to  purchase  life-cycle  supported  OTS  capability.  Measures  are  verification  (yes/no) 
that  RDT&E  is  completed  within  budget;  verification  (yes/no)  planned  capability  delivered  within 
production  and  distribution  budget. 

(3)  Security  plan  achieves  targeted  initial  authorization  and  accreditation  (A&A)  of 
system  security  logical  layer  in  specified  short  period  of  time,  at  specified  low  cost.  Security  plan 
achieves  follow-on  A&A  of  system  changes  faster  and  cheaper  by  inheriting  controls  from  the 
previously  A&A'd  logical  security  layer.  Measures  are  time  and  cost  to  achieve  A&A. 

c.  Business  level  MOP.  Requirement  is  to  satisfy  threshold  levels  of  adaptive  business  process 
level  MOP  such  as  the  following: 

(1)  Competitive  market  of  multiple  vendors  and  COTS  products  exists  and  grows  as  a 
result  of  USG  investment.  Measures  are  numbers  of  traditional  and  non-traditional  Defense 
contractors  who  credibly  compete  for  developmental,  production,  and  sustainment  awards. 

(2)  Cost-per-capability  decreases  as  a  result  of  the  growing  marketplace  of  traditional 
and  non-traditional  solution  providers  incentivized  by  government  investment.  Measure  is 
utility-per-cost  where  "utility"  is  defined  as  "measurable  ability  to  satisfy  requirements"  and 
"cost"  refers  to  lifecycle  costs.  A  measure  of  utility  might  be  percent  improvement  in  a 
designated  MOE  or  MOP. 

(3)  Time-to-value  decreases  as  a  result  of  the  growing  marketplace  of  traditional  and 
non-traditional  solution  providers  incentivized  by  government  investment,  and  improved 
acquisition  processes.  Measure  is  time  between  identification  of  formal  or  informal 
requirement  and  delivery  of  certified,  sustained  (verified  lifecycle  plan),  up-to-date  (verified 
current  version  of  technology),  solution. 

d.  System  level  MOP.  Requirement  is  to  satisfy  threshold  levels  of  system  level  MOP  such  as  the 
following: 


Chris  Gunderson/SAF/AQ  OTI/(o)703-693-4177/(m)831-224-5182/25July2017 


64 


(1)  Engineering  interoperability,  i.e.  ability  to  interchange  components,  improves  as  a 
result  of  the  open  system  approach.  Lagging  measure  is  time  required  to  acquire,  install,  and 
operationally  activate  component  of  interest.  Leading  measures  include  verified  existence  of 
SDK,  verified  compliance  with  open  interface,  verified  existence  of  enterprise  license. 

(2)  Operational  interoperability,  i.e.  ability  to  effectively  share  operational  utilities  such 
as  information,  fuel,  or  ammunition  improves  as  a  result  of  the  open  system  approach. 

Measures  are:  verified  yes/no  that  need-to  share  policy  exists;  verified  that  system  executes 
need-to-share  policy  transactions  according  to  policy;  validated  that  need-to-share  policy  results 
in  improved  operational  MOE. 

(3)  Operational  Availability,  i.e.  assurance  that  system  will  satisfy  enterprise  operational 
requirements  for  mission  dominance  at  any  given  moment,  improves  as  a  result  of  acquisition 
strategy.  Measure  is  the  ratio  of  (effective  "up  time")  divided  by  ((effective  "up  time")  + 
(effective  down  time))  where  (effective  down  time)  =  (time-system-is-broken  +  time-to-repair  + 
time-system-is-out-of-date  +  time-between-obsolescence-and-refresh-of-technology.) 
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APPENDIX  B:  Market  Analysis,  AoA,  Trades,  and  Source  Selection  Matrix 


The  following  graphics  illustrate  a  method  for  developing  an  adaptive  approach  to  market  analysis  and 
associated  source  selection  algorithm.  The  color  code  assigns  green  to  blocks  that  require  a  subjective 
input  value.  Yellow  indicates  a  calculated  value.  "Weights"  are  subjectively  assigned  multiplicative 
factors  that  can  give  more  value  to  one  category  over  another,  or  one  bidder's  approach  compared  to 
other  bidders.'  For  example,  weights  of  1.1  and  0.9  indicate  10%  more  and  less  value,  respectively, 
assigned  to  a  particular  criterion. 

In  this  spreadsheet,  the  weight  factor  subjectively  assigned  to  a  bidder's  overall  approach  is  applied  to 
the  sum  of  all  the  other  adjusted  criteria.  Using  the  prior  example,  a  vendor  with  an  especially  good 
overall  approach  might  receive  a  10%  bump  in  source  selection  score  compared  to  a  vendor  whose 
approach  was  considered  average. 


Adaptive  Acquisition  Market  Analysis,  AoA,  Trades,  and  Source  Selection  Matrix 


"Note:  Weights  may  be  subjectively  assigned  to:  1)  a  success  criteria  in  general,  e.g.  achieving  mission  objective  is  considered 
2Xas  important  as  complying  delivering  on  budget  -  weights  are  2  and  1  respectively;  2)  A  particular  vendor's  past 
performance  is  considered  10%  more  relevant  than  typical  -  prior  performance  relevance  weight  for  that  vendor  =1.1 


Entered  value  1 

Calculated  value 

mhm 

Performance  Category  Success  Criteria 

Measure 

"Weight 

Score 

Overall  approach  weight  S 

factor  o 

ubjective  score  assigned  to 
verall  approach. 

UNSAT  =  0;  SAT  =  1;  Good  = 
1.1;  Great  =  1.2 

N/A 

1 

1 

The  following  panel  describes  mandatory  performance.  A  score  of  "0"  for  any  of  these  criteria  results  in 
a  multiplicative  qualification  factor  of  "0". 


Performance  Category 

Success  Criteria 

Measure 

"Weight 

Score 

Cost  share  or  non-traditional 
partnership 

Willing  to  pay  1/3  costs,  or 

Biddingteam  includes  non- 
traditional  Defense 

contractor. 

yes  =  l;no  =  0 

1 

N/A 

1 

Requisite  security  clearance 

Bidder  has  security  clearance 
required  by  solicitation. 

yes  =  l;  no  =  0 

1 

N/A 

1 

Compliant  with  solicitation 
guidelines 

Bid  meets  nominal 

administrative  and  technical 
requirements  specified  in 
solicitation. 

yes  =  l;  no  =  0 

1 

N/A 

1 

Qualification  factor: 
l=qualified;  0=not  qualified 

Multiplicative  product  of 
qualification  indicators. 

qualified  =  1;  not  qualified 

=  0 
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The  following  "Prior  Performance"  panel  of  the  spreadsheet  describes  criteria  that  intend  to  capture 
elements  of  successful  prior  performance.  It  does  not  provide  an  exhaustive  list,  or  provide  enough 
granularity  to  support  an  objective  weighting  scheme.  Therefore,  it  makes  sense  to  define  a  "good 
enough"  point  of  diminishing  return  on  amount  of  successful  prior  performance.  The  spreadsheet  aims 
to  provide  a  means  to  generally  distinguish  the  group  of  bidders  with  reasonably  significant  relevant 
prior  performance,  from  others.  Strategy  is  to:  1)  set  a  maximum  achievable  adjusted  value  that  any 
firm  with  subjectively  defined  "significant"  prior  performance  is  likely  to  achieve,  and:  2)  subjectively 
assign  a  multiplicative  factor  based  on  perceived  relevance  of  prior  work  to  the  current  requirements. 


Performance  Category 

Success  Criteria 

Measure 

*  Weight 

Score 

Prior  Performance:  Bidder's 
successful  implementations 
in  similar  environments 

Achieved  mission  and/or 
business  objectives 

yes  =  1;  no  =0 

1 

1 

1 

Delivered  on  time 

yes  =  l;  no  =  0 

1 

1 

1 

Delivered  on  budget 

yes  =  1;  no  =0 

1 

1 

1 

Similar  mission  domain 

yes  =  l;  no  =  0 

1 

1 

1 

Verified  conformance  with 
relevant  standard(s)  X 

yes  =  l;  no  =  0 

1 

1 

1 

Verified  tech  refresh 
frequency  (time  between 
updates) 

time  <  technology  half- 
life  =  1;  time>THL  =  0 

1 

1 

1 

Prior  A&A  achieved? 

yes  =  l;  no  =  0 

1 

1 

1 

Has  deployed  in  cloud? 

yes  =  l;  no  =  0 

1 

1 

1 

Other: 

XX  =  1;  YY  =  0 

0 

1 

0 

Other: 

XX  =  1;  YY  =  0 

0 

1 

0 

SUM: 

8 

Max  allowed  value  for 
adjusted  sum  of  prior 
performance  success  criteria 

Criteria  are  representive,  not 
exhaustive  or  granular.  Enter 
point  of  diminishing  return 
where  more  no  longer  = 
better 

Whole  number.  "3"  might 
be  a  reasonable  max 

value. 

MAX: 

3 

Prior  performance  weighting 
factor 

Prior  performance  relevant? 

Not  relevant  =  0;  relevant 
=  1.0;  very  relevant  =  1.1 

N/A 

1 

Prior  Performance  Parameter 

Weighted  sum  of  Prior 
Performance  success  criteria 

scores  times  relevance  factor 

3 

The  "Plug  Test"  panel  below  aims  to  capture  the  measurable  and  testabe  objectives  of  the  acquisition 
together  with  threshold  values.  Intent  is  to  capture  "360°  requirements",  i.e.  not  only  system 
performance  parameters,  but  also  mission  level  and  business  level  measures  of  effectiveness,  and 
acquisition  process  level  measures  of  performance.  In  the  best  case,  program  offices  will  use  actual  test 
tools  to  perform  baseline  validation  and  verification  of  exiting  OTS  capabilities  in  context  with  the  new 
requirements.  In  the  absense  of  actual  test  tools,  USG  will  develop  the  test  tools  and  test  cases  as  part 
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of  the  Systems  Engineering  Plan.  Until  test  tools  are  available,  USG  will  use  best  available  means  to 
perform  objective  evaluation. 

As  offerors  recommend  particular  architectures,  technologies,  products,  services,  and  processes,  and 
the  USG  evaluates  them  against  its  360°  requirements  objectives,  and  refines  them  as  appropriate, 
clarity  of  the  art-of-the-possible  will  emerge.  In  this  sense,  this  approach  lends  itself  to  an  efficient 
iterative  approach  to  market  analysis,  analysis  of  alternatives,  trades,  and  source  selection. 

Again,  the  spreadsheet  allows  the  option  of  entering  a  maximum  possible  Plug  Test  score  in  order  to 
account  for  the  possibility  of  a  point  of  diminishing  return  beyond  which  more  is  not  necessarily  better. 


Performance  Category 

Success  Criteria 

Measure 

^Weight 

Score 

Plug  Test 

Specifically  relevant  OTS 
technology  exits? 

yes  =  1;  no  =  0 

1 

1 

1 

Inherits  security  controls? 

yes  =1;  no  =  0 

1 

1 

1 

Inherits  safety  controls? 

yes  =  1;  no  =  0 

1 

1 

1 

Time  required  to  configure  in 
target  environment? 

time  <  threshold  =  1;  time 

>  threshold  =  0 

1 

1 

1 

Appropriate  SDK  exists? 

yes  =1;  no  =  0 

1 

1 

1 

Enterprise  license  available? 

yes  =  1;  no  =  0 

1 

1 

1 

Verified  mission 
performance,  e.g. 
information  exchanges  occur 
according  to  need-to-share 
policy  &  MOE  achieved 

MOE/MOP  >  threshold  = 
l;MOE/MOP  <  threshold 

=0 

1 

1 

1 

Projected  lifecycle  cost 

$  or  $/time<  threshold  =1; 
$  or  $/time>  threshold  =  0 

1 

1 

1 

Projected  tech  referesh 
frequency  (time  between 
updates) 

time  <  technology  half- 
life  =  1;  time>THL  =  0 

1 

1 

1 

SUM: 

9 

Max  allowed  value  for 
adjusted  sum  of  Plug  Test 
performance  criteria 

Criteria  are  representive,  not 
exhaustive.  Enter  point  of 
diminishing  return  where 
more  no  longer  =  better. 

Whole  number.  "6"  might 
be  a  reasonable  max 

value. 

MAX: 

6 

Plug  Test  Parameter 

Adjusted  weighted  sum  of 
Plug  Test  success  criteria 

scores 

6 

The  spread  sheet  panel  below  calculates  the  overall  source  selection  score  by  applying  the  multiplicative 
weighting  factors  to  the  adjusted,  weighted  sums  of  the  performance  criteria. 

(Overall  approach  factor)  * 

(qualification  factor)  “(prior 
performance  parameter  + 

Source  Selection  Score  Plug  Test  paramenter) 
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Appropriate  Use  of  Non-RDT&E  Appropriations  for  OTA 


26  Oct  2016 


Memorandum  for  the  Record 

Subj:  APPROPRIATE  USE  OF  NON-RDT&E  APPROPRIATIONS  FOR  OTA 

Ref:  (a)  10  USC  2371b 

(b)  OSD  OT  Guide 

(c)  DoD  7000. 14-R,  subj:  Financial  Management  Regulation  (FMR) 


Reference  (a)  provides  authority,  i.e.  "Other  Transactions"  Authority  (OTA),  for  military  services  to 
execute  financial  transactions  that  are  not  generally  subject  to  the  FAR,  in  order  to  conduct  "prototype 
projects."  According  to  ACQuipedia  and  the  OSD  OT  Guide  (reference  (b)),  a  "prototype"  (or  "prototype 
model"  such  as  an  Engineering  Design  Model)  is  "a  physical  or  virtual  model  used  to  evaluate  the 
technical  or  manufacturing  feasibility  or  military  utility  of  a  particular  technology  or  process,  concept, 
end  item,  or  system."  In  this  sense,  prototypes  are  essentially  defined  as  test  artifacts.  According  to  the 
FMR  (reference  (c)),  paying  for  testing  and  prototypes  usually  requires  RDT&E  funds.  Likewise,  the  OT 
Guide  explains  that  since  OTA's  are  designed  to  develop  and/or  evaluate  prototypes,  generally  RDT&E 
appropriations  are  appropriate.  However,  the  FMR  (Volume  2A,  Chapter  1,  paragraph  010213  section  C. 
5.  b.  &  c.)  provides  exceptions  to  the  general  rule.  That  is,  testing  that  is  not  associated  with  RDT&E,  or 
testing  conducted  after  fielding  should  be  financed  with  Procurement  or  O&M  appropriations. 

Thus,  consistent  with  reference  (c),  since  prototypes  are  test  artifacts  by  definition,  prototype  projects 
used  to  support  the  following  activities  should  be  funded  with  O&M  or  Procurement  appropriations: 

1.  Acceptance,  quality  control  and  surveillance  testing  of  articles  obtained  for  other  than  RDT&E 
purposes. 

2.  Routine  testing  in  connection  with  logistic  support. 

3.  Testing  related  to  the  operation  and  maintenance  of  equipment  and  material  acquired  for  use 
under  appropriations  other  than  RDT&E. 

Further,  we  should  use  O&M  or  Procurement  funds  to  acquire  commercial  or  non-developmental  items 
for  testing  and  operational  evaluation  that  do  not  require  RDT&E  engineering,  design  or  integration 
effort.  However,  we  must  use  RDT&E  appropriations  if  the  commercial  item  is  modified  and  requires 
testing  prior  to  approval  for  fielding. 

One  important  potential  application  of  OTA  is  to  conduct  prototype  projects  associated  with  test  and 
evaluation  of  COTS  or  NDI  artifacts  that  may  or  may  not  satisfy  requirements  for  life  cycle  tech  refresh, 
or  recapitalization  of  existing  deployed  systems.  Lifecycle  tech  refresh  is  an  O&M  activity  (not  RDT&E). 
Re-capitalization  is  a  procurement  activity  (not  RDT&E).  OTA  prototype  projects  may  aim  to  encourage 
vendors,  including  non-traditional  Defense  contractors,  to  use  their  own  internal  research  efforts  to 
evolve  off-the-shelf  products  that  simply  "plug  into"  the  fielded  architecture.  No  engineering,  design,  or 
integration  effort  would  be  required  to  field  such  off-the-shelf  capability.  Thus,  it  is  appropriate  to 
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execute  the  following  classes  of  prototype  projects,  which  would  be  used  to  evaluate  the  suitability  of 
COTS/NDI  products  for  tech  refresh  or  recapitalization,  under  OTA,  with  O&M  or  Procurement 
appropriations  respectively. 

1.  COTS/NDI  products  for  use  in  testing  in  support  of  tech  refresh  or  re-capitalization 

2.  Acquisition  process  models  used  in  support  of  lifecycle  tech  refresh  or  re-capitalization 

3.  Test  and  evaluation  products  and  services  used  for  testing  of  COTS/NDI  for  tech  refresh  or  re¬ 
capitalization. 

However  in  classes  of  prototype  projects  described  above,  the  government  customer  must  field  the 
refreshed  technology  out-of-the-box,  i.e.  with  no  further  acceptance  testing  or  certification  required. 
This  caveat  forces  the  government  customer  to  assure  themselves  that  their  test  and  evaluation 
methodology  is  a  robust  means  to  validate  that  their  tech  refresh  requirements  are  correctly  posed,  and 
verify  that  vendor  claims  to  satisfy  them  with  off-the-shelf  capability  are  true. 

In  this  sense,  General  Counsel  and/or  Financial  Management  authorities  must  agree  that  testing  is,  in 
cases  of  COTS/NDI  tech  refresh  or  re-capitalization,  a  generic  service  for  evaluating  suitability  of 
COTS/NDI  for  direct  fielding.  In  those  cases,  government  O&M  or  Procurement  appropriations  for 
prototype  projects  associated  with  testing  may  not  be  used  to  support  government-funded 
development.  However,  it  would  be  appropriate  for  commercial  vendors  to  voluntarily  spend  Internal 
R&D  funds  as  a  result  of  GFI  test  results.  Thus  a  GFI  test  service  incentivizes  industry  to  invest  to  evolve 
products  that  are  suitable  to  plug  directly  into  fielded  government  systems,  and  therefore  achieve 
lifecycle  tech  refresh  and  recapitalization  in  step  with  the  pace  of  COTS  evolution. 
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OSD  Other  Transactions  Guide  for  Prototype  Projects 


Other  Transactions 
Guide 
for 

Prototype  Projects 


January  2017 
(Version  1 .2.0) 
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Appendix  B:  Measuring  Value,  and  Managing  Risk 


USAF  investments  in  weapon  systems  or  other  capabilities  must  return  value,  where  value  is  measured 
objectively  in  terms  of  military  effectiveness,  and  in  terms  of  creation  of  wealth  within  the  U.S. 
economy.  The  taxpayers  deserve  both  an  effective  defense  force  and  economic  growth  as  a  result  in 
their  investment  in  the  USAF.  Moreover,  the  USAF  must  deliver  value  to  the  War  Fighters  in  order  to 
address  continuing  and  emerging  threats.  The  current  status  quo  is  unsatisfactory  in  both  regards  since 
the  DoD  acquisition  system  has  been  plagued  by  cost  and  schedule  overruns. 

The  status  quo  approach  to  measuring  USAF  programs  captures  cost  overruns  and  schedule  delays  on  its 
programs  to  assess  performance.  However,  it  does  not  provide  the  leading  indicators  and  incentives 
necessary  to  effect  course  changes  that  would  have  precluded  them.  Indeed,  the  solution  to  the 
program  managers'  dilemma  is  to  re-baseline  budget  and  schedule,  generally  without  considering 
resulting  degradation  to  the  value  of  the  program's  original  value  proposition.  To  address  this  issue,  the 
DoD  has  implemented  several  acquisition  reform  policies  but  none  have  been  generally  successful. 

Our  approach  to  address  these  issues  is  to  create  a  new  metric  for  Return  on  Investment  (Rol)  focused 
on  value.  Generally,  Rol  for  a  Defense  system  is:  operational  utility  delivered;  per  lifecycle  monetary 
cost  of  the  utility;  per  time  increment  required  to  turn  a  monetary  investment  into  utility.  Utility  in  this 
context  is  a  measurable  ability  to  satisfy  a  requirement.  So,  units  of  utility  might  be  those  of  time, 
spatial  dimension,  probability,  environmental  factors,  or  quantity  of  good  or  bad  things.  Units  of  utility 
might  also  be  percent  change,  good  or  bad,  measured  against  a  specified  baseline.  In  the  sense  of  risk 
vs.  reward,  "reward"  is  equivalent  to  the  ROI  earned  via  developing  an  increment  of  system-enabled 
capability. 


Rol  =  V(u,  c,t)  =  u(t)  x  (c(t))-1  x  f(td-1)  (1) 

Rol  =  Return  on  Investment 

V(u,c,t)  =  Value  of  a  system,  or  component(s)  thereof,  as  a  function  of  u,  c,  and  t 

u(t)  =  Utility  of  a  system,  or  component(s)  thereof,  as  a  function  of  time 

c(t)  =  Monetary  cost  of  a  system,  or  component(s)  thereof,  as  a  function  of  time 

f(td-l)  =  Function  of  the  time  it  takes  to  design,  develop,  test,  certify,  and  deploy  an  increment 

of  utility  wherein  value  is  inversely  proportional  to  elapsed  time. 


Because  utility  is  equivalent  to  the  degree  to  which  a  capability  satisfies  requirements,  u  can  be 
measured  and  modeled  the  same  way  requirement  satisfaction  is  measured  or  modeled.  Consistent 
with  established  systems  engineering  best  practice,  we  define  measures  of  effectiveness  (ME)  as  lagging 
indicators,  i.e.  objective  parameters  that  describe  operational  outcomes.  We  define  measures  of 
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performance  (MP),  as  leading  indicators,  i.e.  objective  parameters  that  describe  important  attributes  of 
system  or  process  outputs  that  are  only  important  if  they  lead  demonstrably  to  greater  utility,  and  thus 
Rol.  (Roedler  &  Jones,  2005), 

Typical  ME  for  systems  include: 

•  Probabilities  of  achieving  desired  outcomes 

•  Time  required  for  completing  tasks 

•  Numbers  of  good  or  bad  things  that  happen  as  a  result  of  processing  information 

•  Proficiency  scores 

•  Percent  change  associated  with  the  above 
Typical  MP  for  EIS  include: 

•  Latencies 

•  Reliability,  Availability,  and  Maintainability  (RAM) 

•  Standard  compliance 

•  Capacity 

•  Precision 

•  Size  weight  and  power 

•  Percent  change  associated  with  the  above 


Note  that  the  distinction  between  ME  an  MP  can  in  some  cases  depends  on  where  the  systems 
boundaries  are  drawn.  The  ME  for  an  upstream  system  process  might  be  an  MP  for  a  downstream 
system  process.  For  example,  Probability  of  Detection  might  be  an  ME  for  an  upstream  military 
surveillance  subsystem,  but  an  MP  for  a  downstream  target  selection  subsystem,  the  ME  of  which  is 
Probability  of  Interdiction. 

Risk-reward  optimization  factors,  (Rx)  are  the  most  significant  contributors  and/or  detractors  to 
achieving  critical  system-level,  or  process-level,  performance  characteristics  X.  Consistent  with  best 
practices  for  investments,  we  select  appropriate  Rx  and  then  define  MP  and  ME  to  align  with  them. 
Heuristic  analysis  of  historical  military  system  success  and  failure  cases  suggests  the  following  examples 
of  typical  Rx: 
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Ro  =  Ability  to  continuously  capture  the  operational  customers'  perception  of  value 
within  rapidly  evolving  operational  domains  (e.g.  by  establishing  continuous  feedback 
loop.)  An  Mp might  be  "customer  contact  hours."  i.e.  a  measure  of  developers' 
performance  in  communicating  with  the  customer,  and  a  leading  indicator  of 
developers'  ability  to  achieve  greater  utility  in  the  eyes  of  the  customer.) 

Rt  =  Ability  to  continuously  harvest  technological  value  in  rapidly  evolving  technological 
domains  (e.g.  by  applying  best  commercial  practices  for  open  standard  product  line 
architecture.)  An  MPmight  be  "time  required  to  configure  component  in  the  EIS 
technical  environment,"  which  might  be  a  leading  indicator  of  the  ME  "time  it  takes  to 
perform  an  increment  of  tech  refresh.") 

R$  =  Ability  to  predict  lifecycle  costs  for  continuously  evolving  capability  (e.g.  by  heavily 
leveraging  existing  off-the-shelf  technologies  that  come  with  well-established  life  cycle 
tech  refresh  cost  models.  MP  might  be  "lifecycle  costs  are  known  and  are  less  than  'X'.") 

Ria  =  Ability  to  balance  the  need-to-protect  information  and  (e.g.)  network  resources, 
with  the  need-to-share  them  across  security  domains  (e.g.  by  implementing  need-to- 
share  and  need-to-protect  with  high  assurance  virtual  technology.)  An  MP/E  might  be 
"accredited  ability  to  execute  dynamic-policy-based  need-to-share  decision." 

Rvi  =  Ability  to  find  and  deliver  valued  information  bits  within  tightly  constrained 
decision  windows,  given  large  and  growing  backdrop  of  available  information  bits  (e.g. 
by  identifying  critical  conditions  of  interest  and  implementing  automated  "smart  push" 
alerts.  MP/E  might  be  "run  time  demonstration  of  decision  cycle  time  compression 
against  use  case  of  interest.") 

RPs  =  Availability  of  professional  skills  required  for  adaptive  evolutionary  development 
(e.g.  by  evaluating  vendors'  prior  performance  against  similar  adaptive  evolutionary 
system  development.  MP  might  be  "documented  success  in  prior  performance  on 
similar  open  system  project.") 


Efficient  systems  and  processes  should,  by  design,  facilitate  effective  outcomes.  That  is,  as  new 
technology  is  deployed,  the  system  and  process  efficiencies  should  improve,  and  the  operational 


effectiveness  should  also  improve  predictably  as  a  result.  Here  efficiency  is  defined  as 


Useful  Output 
Intput 


)■ 


,  ..  ,  , Useful  Outcome , 

Effectiveness  is  defined  as  ( - ). 

Intput 


Hence  tested  values  of  ME,  which  describe  outcomes, 


should  be  highly  mathematically  correlated  to  tested  values  of  MP,  which  describe  outputs.  Thus,  the 
correlation  coefficient  (ppu)  of  the  leading  performance  indicator  MP,  and  lagging  utility  indicator  ME, 
must  be  greater  than  zero.  This  is  simply  a  mathematical  expression  of  the  definition  of  leading  and 
lagging  indicators,  namely  that  the  leading  measure  of  output  actually  has  skill  at  predicting  the  lagging 
measure  of  outcome.  Ideally  the  correlation  coefficient  for  leading  and  lagging  indicators  should 
approach  1.0,  i.e.  the  leading  measure  of  performance  should  ideally  perfectly  predict  targeted 
outcomes 
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Accordingly,  modeling  or  testing  methods  should  validate  and  verify  the  hypothesis  that  "if  the  EIS  MP 
collectively  improve  -  through  appropriate  management  of  risk-reward  factors  —  then  the  EIS  ME  will 
also  improve."  "Validation"  means  confirmation  that  the  ME  actually  effectively  describes  outcomes 
valued  by  the  customer.  "Verification"  means  confirmation  that  the  selected  performance 
requirements,  expressed  as  MP,  are  satisfied,  and  that  satisfaction  of  the  performance  requirements 
result  in  improved  ME. 


(2) 


RT  =  Threshold  requirement  for  system  Rol 
RO  =  Objective  requirement  for  system  Rol 

ppo  =  Correlation  coefficient  of  system-level  and  process-level  performance  and  operational- 
level  performance 

opo  =  Covariance  of  system-level  and  process-level  performance  and  EIS  functional  test  results 
(normalized  across  family  of  tests) 

op  =  Standard  deviation  of  performance  tests  (normalized) 
oo  =  Standard  deviation  of  operational  tests  (normalized) 


Again,  equation  (2)  is  simply  a  mathematical  expression  suggesting  that  the  test  plan  for  system 
development  projects  should  validate  and  verify  that  the  parameters  chosen  to  predict  progress  against 
achieving  targeted  Rol  i.e.  manage  risk,  actually  do  so.  It  is  a  mathematical  re-statement  of  what  PMI's 
"quantitative"  approach  to  risk  management  should  achieve,  and  what  risk  models  for  highly  assured 
system  engineering  processes,  and  successful  financial  portfolios,  actually  achieve. 

Calculation  of  Probability  of  Success 

Traditional  program  management  doctrine  suggests  thoroughly  framing  and  managing  risks  in  terms  of 
impact  to  cost,  schedule,  and  performance.  PMI  suggests  also  categorizing  positive  risks  in  terms  of 
potential  good  consequences,  and  the  probability  of  their  occurrence.  However,  in  practice,  PMs  tend 
to  focus  on  probability  and  consequence  of  negative  risk.  Here  we  suggest  a  fundamental  shift  by 
offering  a  mathematical  approach  for  focusing  on  the  desired  good  outcome,  namely  Rol.  In  general, 
the  probability  of  achieving  the  threshold  requirement  for  Rol  (P(Vt))  is  equal  to  the  joint  probability  of 
achieving  threshold  requirements  for  cost  (ct),  performance  (pt),  and  schedule  (st): 


P  [Vt]  =  P(ct  nut  n  st) 
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If  the  probabilities  are  independent  of  each  other,  then: 


(3) 


P[Vt]  =  P[ct]*P[ut]*P[st] 

P[Vt]  =  Probability  of  achieving  threshold  level  of  valued  outcome,  i.e.  Rol. 

P(ct  flUjfl  st)  =  Joint  probability  of  achieving  threshold  level  of  monetary  budget,  utility,  and  schedule 
requirements 

P[ct]  =  Probability  of  satisfying  threshold  level  of  monetary  budget  requirements. 

P[ut]  =  Probability  of  achieving  threshold  level  of  utility  requirements. 

P[st]  =  Probability  of  achieving  threshold  level  of  schedule  requirements. 


The  PM's  task  is  to  evolve  algorithms  that  correlate  risk  mitigating  input  behaviors,  to  the  probability  of 
achieving  measurably  successful  process  outputs  and  outcomes.  Given  these  models  of  risk/rewards 
probability  and  impact,  PM's  can  credibly  optimize  acquisition  process  effectiveness  using  tools  such  as 
traditional  risk  matrices.  (See  figure  3.) 
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Risk  Management 


.  .  P[Vt]  =  P  ([cj,  [ut],  [sj) 

Assuring  high 

payoff  is  as 

important  as 

assuring  low  risk 


Likelihood  Rationale: 

5  Almost  Certain 
4.  Probable 
3.  50/50 
2.  Improbable 
1 .  Almost  No 
Chance 


Rol  =  u/c  X  l/td 


Consequence  Rationale: 

5.  Risk:  >=  100% 
degradation* , -Reward:  >= 
100%  improvement* 

4.  +/-  80% 

3.  +/-  00% 

2.  +/-  40% 

1.  Risk  <=  20% 
degradafon.'Reward:  <= 
20%  improvement 


^  V»r«  W  •  &  w  t  ywafc-'n  J  xss'rxt 

Target  technologies/processes  with  high  reward 

potential 

•  Operators  Identify  critical  mission  threads  and  associated 
desired  outcomes  up  front 

•  Establish  associated  testable  Measures  of  Effectiveness 
(MOE)  lag  metrics 

•  Establish  Measures  of  Performance  (MOP)  lead  metrics 
that  are  testably  coupled  to  MOE  lag  metrics 

•  Build  iterative  test  plan  that  assures  MOP  lead  metrics 
and  MOE  lag  metrics 

•  Perform  AoA  of  potential  technology  components  per  the 
above 


Target  technology  portfolio  with 

balanced  risk  profile 

•  At  least  80%  of  technology 
components  must  exist  as 
COTS/GOTS* 

•  Any  developed  technology  has  known 
transition  path  to  COTS/GOTS 

•  All  performers  have  prior  success  with 
Open  System  development 

•  Project  scope  and  process  must 
support  technology  onboarding  within 
"Moore's  Lav;"  time  window 

*COTS/GOTS=  configurable  out  of  the  box  via  open 
standards  and  comes  with  known  intellectual  property 
rights  and  life  cycle  support  model 


Figure  3:  Optimizing  Rol  depends  on  modeling  and/or  measuring  co-dependence  and  evolution  of  risk  and  reward 

Cost  Probability  Model 

If  "cost"  is  defined  as  monetary  lifecycle  investments  for  developing,  testing,  evaluating,  certifying, 
deploying,  maintaining,  and  upgrading  a  system  or  component(s)  thereof;  then  the  probability  of 
achieving  threshold  targets  for  cost  depend  on  optimizing  the  combination  of,  for  example:  upfront 
costs  including  initial  purchase  and  any  required  infrastructure  investments;  accuracy  of  projected 
upgrade  and  maintenance  costs;  and  the  anticipated  lifetime  of  the  system.  For  example: 


P[ct)  *(.AC0=  ^=l£k)  (4) 


(ca  -  ce) 


2 

Upfront 

investments 


+  (ca  -  O 


2 

Developmental 
Test  and  Cert  costs 


+  (ca  -  ce) 


2 

Maintenance 
and  Upgrade  costs 


+  (ca  -  Ce) 


2 

nth  other  cost 


n 


P[ct]  =  Probability  of  achieving  threshold  requirement  for  cost 
(Aco  =  =  Availability  of  cost  objectives 
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Ce  =  Previously  estimated  total  system  lifecycle  costs  including  upfront  costs  for  infrastructure 
and  initial  purchases,  engineering  costs,  and  lifecycle  upgrade  and  maintenance  costs. 

(Tee  =  Root  mean  square  error  of  actual  lifecycle  costs  vs.  estimated  costs 

ca  =  Actual  costs  for  the  ")  indicated"  activity 

ce  =  Previously  estimated  costs  for  the  ")  indicated"  activity 

Utility  Probability  Model 

Here  we  suggest  that  the  probability  of  achieving  the  targeted  level  of  utility  depends  on,  for  example, 
the  quality  of  requirements,  scope  of  the  potential  solution  space,  efficiency  of  the  Analysis  of 
Alternatives  (AoA),  efficiency  of  the  capability  integration  platform,  and  the  quality  of  Test  and 
Evaluation  (T&E.) 

Achieving  sufficient  quality  of  requirements  demands  a  process  that  provides  objective  feedback  from 
the  operational  customer  community  several  times  during  any  particular  developmental  cycle.  Achieving 
sufficient  scope  of  solution  set  demands  a  process  that  socializes  the  system  project  use  cases  broadly 
across  the  landscape  of  innovative  industry.  Achieving  efficiency  of  AoA  requires  an  automated  process, 
objective  measures,  and  incentives  to  allow  and  encourage  solution  providers  to  self-demonstrate  the 
Vp  of  their  offerings.  Achieving  efficiency  in  the  integration  platform  requires  well-defined  architectural 
functions  and  open  standard  interfaces.  Achieving  quality  of  T&E  requires  test-based  designs,  persistent 
test  frameworks,  and  iterative  testing  throughout  project  execution. 

To  maximize  the  probability  of  satisfying  system  threshold  utility  requirements,  the  project  work 
breakdown  should  scrupulously  allocate  the  proper  relative  proportions  of  billable  time  spent: 
processing  operational  customers'  feedback;  evaluating  evolving  capabilities  in  the  market;  carefully 
rationing  any  time  spent  developing  immature  technologies;  and  testing;  etc.  The  project  manager 
should  adjust  this  schedule  optimization  model  at  each  successive  developmental  cycle.  Assuming  that 
process  is  ongoing,  PMs  can  model  the  probability  of  satisfying  threshold  levels  of  efficiency  and 
effectiveness  by  tracking  both  whether  the  critical  activities  occurred  as  scheduled,  and  how  well  the 
test  scores  aligned  with  targeted  measures.  If  the  right  risk/reward  optimization  activities  are  scheduled 
and  performed,  test  results  should  both  improve  and  become  more  predictable  as  the  project 
progresses.  For  example: 


P[ut]  oc  (Aca  =  t-^)  (5) 


2 

'nt  other  critical 


activitiy 


n 
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P[ut]  =  Probability  of  achieving  threshold  requirement  for  effectiveness/performance/utility 

( Aca  =  td  ^ca)  =  Availability  of  critical  scheduled  activities 

td 

td  =  Originally  scheduled  time  for  designing,  engineering,  T&E,  and  certification  of  an 
incremental  system  capability  delivery. 

Oca  =  Root  mean  square  error  of  actual  time  spent  on  critical  risk-reward  optimization  activities 
compared  to  originally  scheduled  time  for  those  activities. 
ta  =  Time  actually  spent  performing  the  ")  indicated"  risk-reward  optimizing  activity 
ts  =  Time  originally  scheduled  for  the  ")  indicated"  risk-reward  activity 


Schedule  Probability  Model 

This  formulation  of  the  threshold  value  of  "schedule"  aims  to  assure  that  capability  is  designed, 
developed,  tested,  certified,  delivered  and  refreshed  within  the  "technology  half-life"  of  the  capability  in 
question.  The  concept  of  technology  half-life  recognizes  the  value  of  any  unit  of  IT  is  highly  perishable. 
Technology  half-life  is  the  length  of  time  it  takes  for  the  value  of  the  IT  unit  of  interest  to  decrease  to 
notionally  half  of  its  original  value.  In  practice,  determining  technology  half-life  is  usually  subjective. 

The  goal  is  to  deploy  the  technology  standard  of  interest  no  later  than  midway  through  its  optimally 
useful  lifetime. 


Achieving  assurance  of  "schedule  value"  requires  a  schedule  process  that  standardizes  and  parallelizes 
sub  process,  e.g.  testing  part  A  while  with  developing  part  B;  de-conflicts  resources,  e.g.  schedules 
enterprise  testing  resources  across  independent  sub  tasks;  schedules  work  to  include  preparing 
independently  useful  capability  modules  that  can  be  developed  and  or  procured  and  deployed 
irrespective  of  schedule  delays  associated  with  other  modules. 


P  [st]  <x  ( Adv 


_  Zn=  1  (W  fn) 

1 K n  (WPn) 


)  (6) 


P[st]  =  Probability  of  achieving  threshold  schedule  requirements 

Adv  =  Availability  of  developed  value.  I.e.  weighted  sum  of  completed  work  units  divided  by 
weighted  sum  of  scheduled  work  units. 

Wfn  =  Successfully  completed  work  unit. 

Wpn  =  Scheduled  work  unit. 

Kn  =  Weighting  factor.  Weighting  should  take  into  account  a  clear  delineation  of  how  any  work 
unit  relates  to  project  critical  path, 
n  =  Counting  index 

f  =  Number  of  successfully  completed  and  tested  scheduled  work  units. 
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p  =  Number  of  scheduled  work  units. 
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Appendix  C:  Sample  Authorities  for  Adaptive  Acquisition 


DODI  5000.02 

"...  The  models  provide  baseline  approaches.  A  specific  program  should  be  tailored  to  the  unique 
character  of  the  product  being  acquired...  acquisition  management ...  (traditional  or)  tailored  variation  ... 
depends  on  ...  knowledge  about  the  capability ...  and  risks  and  costs ...  to  support  a  sound  business 
decision  to  proceed  to  the  next  phase..." 

Enel.  6: 

"...  Employ  effective  performance-based  logistics  (PBL)  planning,  development,  implementation,  and 
management  in  developing  a  system's  product  support  arrangements.  PBL  is  performance-based 
product  support,  where  outcomes  are  acquired  through  performance-based  arrangements  that  deliver 
warfighter  requirements  and  incentivize  product  support  providers  to  reduce  costs  through 
innovation..." 

From  November  22,  2013  ASD  (L&MR)  "Performance  Based  Logistics  Comprehensive  Guidance" 
Memo 

"PBL  is  synonymous  with  performance  based  life  cycle  product  support,  where  outcomes  are  acquired 
through  performance  based  arrangements  that  deliver  Warfighter  requirements  and  incentivize  product 
support  providers  to  reduce  costs  through  innovation.  These  arrangements  are  contracts  with  industry 
or  inter-governmental  agreements.  Attributes  of  an  effective  PBL  arrangement  include: 

•  Objective,  measurable  work  description  that  acquires  a  product  support  outcome. 

•  Appropriate  contract  length,  terms,  and  funding  strategies  that  encourage  delivery  of 
the  required  outcome. 

•  A  manageable  number  of  metrics  linked  to  contract  requirements  that  reflect  desired 
Warfighter  outcomes  and  cost  reduction  goals. 

•  Incentives  to  achieve  required  outcomes  and  cost  reduction  initiatives. 

•  Risks  and  rewards  shared  between  government  and  commercial  product  support 
integrators  and  providers. 

•  Synchronization  of  product  support  arrangements  to  satisfy  Warfighter  requirements." 


"Other  Transactions"  Authority 

•  10  USC  section  2371-  Research  projects:  transactions  other  than  contracts  and  grants 

•  10  USC  section  2371b  —  Authority  of  the  Department  of  Defense  to  Carry  on  Certain  Prototype 
projects 

•  FY16  NDAA  Section  815  —  expansion  of  10  USC  section  2371b:  Authority  of  the 

Department  of  Defense  to  Carry  on  Certain  Prototype  projects.  DoD  officials  may  use 
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"Other  Transactions"  Authority  for  funding  prototypes  of  solutions  that  may  be  acquired 
or  developed  to  improve  military  capability  per  the  following: 

•  Limit  of  $50M-$250M  per  prototype  unless  higher  ceiling  approved  by  AT&L 

•  At  least  one  of  the  following  criteria  are  met: 

•  A  least  one  nontraditional  defense  contractor  participates 

•  All  significant  non-government  participants  are  small  businesses  or 
nontraditional 

•  At  least  one  third  of  the  total  cost  provided  by  commercial  partner 

•  SAE  justifies  in  writing  that  OTA  allows  innovative  business 
arrangements  not  feasible  or  appropriate  under  a  contract,  or 
expansion  of  defense  supply  base  not  practical  or  feasible  under  a 
contract. 

•  To  maximum  extent  practical,  "competitive  procedures"  shall  be  used  to  award 
OTA 

•  Prototype  provides  for  award  of  follow  on  production  transaction  or  contract  if 
and  only  if: 

•  "Competitive  procedures"  were  used  for  award  of  OTA 

•  Participants  in  the  OTA  "successfully  complete"  the  prototype 


JCIDS  Manual  Appendix  B 

For  sustainment  of  previously  fielded  capability  solutions,  a  new  ICD,  CDD,  or  CPD  is  not  required  to 
retain  or  restore  capabilities  or  perform  technology  refresh  of  fielded  systems  that  have  a  validated 
Operational  Requirements  Documents  (ORD),  ICD,  CDD,  or  CPD.  For  example,  subsystems  that  have 
approved  performance  parameters  but  are  no  longer  able  to  meet  those  parameters  can  be  updated  or 
replaced  to  meet  production  threshold/objective  values  under  the  authority  of  the  previously  validated 
capability  requirement  document.  However,  if  the  MDA  or  other  decision  maker  requires  the  capability 
document(s)  be  "revalidated"  prior  to  supporting  additional  production  or  technology  refresh,  the 
legacy  documents  shall  be  transcribed  into  current  document  formats  and  content  prior  to  submitting 
for  review  and  validation. 

FY  16  NDAA  Section  803:  Expansion  of  Rapid  Acquisition  Authority 

In  case  of  urgent  need,  SECDEF  may  appoint  a  senior  acquisition  official  who,  with  written  justification: 

•  May  waive  any  laws  and  regulations  in  order  to  satisfy  requirement 

•  May  use  any  color  of  money,  not  to  exceed  $200M/year  total 

•  Shall  attempt  to  award  contract  within  15  days 

•  Shall  transition  to  normal  acquisition  process  within  2  years 

FY  16  NDAA  Section  804:  Middle  Tier  of  Acquisition  for  Rapid  Prototyping  and  Rapid  Fielding 

AT&L  shall  provide  guidance  for  two  rapid  acquisition  paths: 
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•  RAPID  PROTOTYPING: ...  use  of  innovative  technologies  to  rapidly  develop  fieldable  prototypes 
to  demonstrate  new  capabilities  and  meet  emerging  military  needs...  prototype  ...  demonstrated 
in  an  operational  environment ...  residual  operational  capability  within  five  years.... 

•  Merit-based  process  to  evaluate  innovative  technology 

•  Acquisition  strategies  for  the  program 

•  Cost-sharing  process  with  the  military  departments 

•  Demo/evaluation  process 

•  Transition  process 

•  OSD  will  provision  rapid  prototyping  fund 

•  RAPID  FIELDING....  use  of  proven  technologies  to  field  production  quantities  of  new  or  upgraded 
systems  with  minimal  development  required. ...  begin  production  within  six  months  and 
complete  fielding  within  five  years... 

•  Merit-based  process  for  evaluating  existing  products/proven  technologies 

•  Demo/evaluation  process 

•  Acquisition  strategy 

•  Process  for  logistics  support  and  system  interoperability 

FY  16  NDAA  Section  805:  Use  of  Alternate  Acquisition  Paths  to  Acquire  National  Security 
Capabilities 

SECDEF  shall  establish  alternative  acquisition  pathways  that  shall: 

•  Be  separate  from  existing  acquisition  procedures 

•  Streamline  contracting,  budgeting,  and  requirements  processes 

•  Base  alternative  acquisition  paths  on  urgency  and  nature  of  capability 

•  Maximize  use  of  flexible  authorities  in  existing  law  and  regulation 

FY16  NDAA  Section  806:  SECDEF  Waiver  of  Acquisition  Laws  to  Acquire  Vital  National  Security 
Capabilities 

SECDEF  may  waive  and  law  or  regulation  regarding: 

•  Requirement  or  specification 

•  RDT&E 

•  Production,  fielding,  and  sustainment 

•  Solicitation,  source  selection,  and  award  of  contracts 

If  and  only  if: 


•  Capability  is  vital  to  national  security 

•  Waived  law/reg  would  impede  effective  acquisition  of  the  capability 

•  Purpose  of  waived  law/reg  can  be  addressed  alternatively 

Defense  Acquisition  Guide 

7.10.1.  The  Impetus  for  COTS  Software  Solutions 
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One  of  the  Department's  goals  is  to  migrate  to  COTS  solutions  to  fill  Information 
Technology  capability  gaps. 

Subtitle  III  of  Title  40  of  the  United  States  Code  (formerly  known  as  Division  E  of  the 
Clinger-Cohen  Act  (CCA)  (referred  to  as  "Title  40/Clinger-Cohen  Act")  and  DoD 
Instruction  5000.02,  Enclosure  2,  paragraphs  4.c.(6)  and  5.d.(l)(b)3,  all  require  the  use 
of  COTS  IT  solutions  to  the  maximum  practical  extent. 

7.10.2.  Definition 

Commercial,  Off-the-Shelf  (COTS)  is  defined  as  "commercial  items  that  require  no 
unique  government  modifications  or  maintenance  over  the  life  cycle  of  the  product  to 
meet  the  needs  of  the  procuring  agency." 

[From  the  Twelfth  Edition  of  GLOSSARY:  Defense  Acquisition  Acronyms  and  Terms.) 

7.10.3.  Mandatory  Policies 

The  following  bullets  quote  or  paraphrase  sections  in  the  DoD  5000  series  that 
specifically  address  COTS: 

DoD  Directive  5000.01,  "The  Defense  Acquisition  System"  Paragraph  El. 1.18,  states 
"...The  DoD  Components  shall  work  with  users  to  define  capability  needs  that  facilitate 
the  following,  listed  in  descending  order  of  preference: 


"El. 1.18.1.  The  procurement  or  modification  of  commercially  available 
products,  services,  and  technologies,  from  domestic  or  international 
sources,  or  the  development  of  dual-use  technologies; .... " 


Hence,  commercially  available  products,  services,  and  technologies  are  a  first  priority  for 
acquisition  solutions. 

DoD  Instruction  5000.02,  "Operation  of  the  Defense  Acquisition  System" 

•  DoD  Instruction  5000.02,  Enclosure  2,  paragraph  4.c.(6),  states  that  "existing 
commercial  off-the-shelf  (COTS)  functionality  and  solutions  drawn  from  a 
diversified  range  of  large  and  small  businesses  shall  be  considered,"  when 
conducting  the  Analysis  of  Alternatives. 

•  Enclosure  5,  "IT  Considerations,"  Table  8,  "Title  40,  Subtitle  Ill/CCA  Compliance 
Table,"  requires  that,  to  be  considered  Title  40/CCA  compliant,  the  Department 
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must  redesign  the  processes  being  supported  by  the  system  being  acquired,  to 
reduce  costs,  improve  effectiveness  and  maximize  the  use  of  COTS  technology. 

•  Enclosure  5,  "IT  Considerations,"  Section  8,  states  that:  "When  the  use  of 
commercial  IT  is  considered  viable,  maximum  leverage  of  and  coordination  with 
the  DoD  Enterprise  Software  Initiative  shall  be  made." 

7.10.4.  COTS  Software-Reuse  Custom  Components 

Modifying  the  core  code  of  a  COTS  product  should  be  avoided.  It  is  possible  to  add  code 
to  the  existing  product,  to  make  the  product  operate  in  a  way  it  was  not  intended  to  do 
"out-of-the-box."  This,  however,  significantly  increases  program  and  total  life-cycle 
costs,  and  turns  a  commercial  product  into  a  DoD-unique  product.  The  business 
processes  inherent  in  the  COTS  product  should  be  adopted,  not  adapted,  by  the 
organization  implementing  the  product.  Adopting  a  COTS  product  is  done  through 
business  process  reengineering  (BPR).  This  means  the  organization  changes  its  processes 
to  accommodate  the  software,  not  vice  versa.  In  many  cases  there  will  be  a  few 
instances  where  BPR  is  not  possible.  For  example,  due  to  policy  or  law,  it  may  be 
necessary  to  build  or  acquire  needed  reports,  interfaces,  conversions,  and  extensions.  In 
these  cases,  adding  to  the  product  must  be  done  under  strong  configuration  control.  In 
cases  where  a  particular  COTS  product  does  not  provide  the  entire  set  of  required 
functionality,  a  "bolt-on"  could  be  used.  A  bolt-on  is  not  part  of  the  COTS  software 
product,  but  is  typically  part  of  a  suite  of  software  that  has  been  certified  to  work  with 
the  product  to  provide  the  necessary  additional  functionality.  These  suites  of  software 
are  integrated  to  provide  the  full  set  of  needed  functionality.  Using  a  bolt-on,  however, 
also  increases  program  and  total  life-cycle  costs. 

See  section  7.10.6.3  for  a  more  detailed  discussion  of  reports,  interfaces,  conversions, 
and  extensions. 

7.10.5.  COTS  Integration  into  the  Acquisition  Life  Cycle 

The  actions  below  are  unique  to  acquiring  COTS  Information  Technology  solutions. 

These  activities  should  occur  within  a  tailored,  responsive,  and  innovative  program 
structure  authorized  by  DoD  Instruction  5000.02.  The  stakeholder  primarily  responsible 
for  each  action  is  shown  at  the  end  of  each  bullet. 

7.10.5.1.  Before  Milestone  A 

•  Define  strategy  and  plan  for  conducting  BPR  during  COTS  software 
implementation  phase  of  the  program. 

(Sponsor/Domain  Owner) 
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•  Consider  COTS  and  BPR  when  developing  the  Analysis  of  Alternatives.  (See 
section  3.3  and  Table  7.8.4.T1  of  this  guidebook). 

(Sponsor/Domain  Owner) 

•  Consider  commercially  available  products,  services,  and  technologies  when 
defining  initial  user  needs  in  the  Initial  Capabilities  Document. 

(Sponsor/Domain  Owner) 

•  When  developing  the  Technology  Development  Strategy  and/or  the  Acquisition 
Strategy,  consider  commercial  best  practice  approaches  and  address  the 
rationale  for  acquiring  COTS. 

(Program  Manager  (PM)) 

7.10.5.2.  Before  Milestone  B 

•  To  the  maximum  extent  possible,  redesign  business  processes  to  conform  to  the 
best  practice  business  rules  inherent  in  the  COTS  product.  Define  a  process  for 
managing  and/or  approving  the  development  of  reports,  interfaces,  conversions, 
and  extensions. 

7.10.5.3.  Before  Milestone  C  or  Full  Rate  Production  Decision/Full  Deployment 
Decision  Review 

•  Ensure  scope  and  requirements  are  strictly  managed  and  additional  reports,  interfaces, 
conversions,  and  extensions  objects  are  not  developed  without  prior  authorization. 
(Program  Manager  (PM)) 

•  Ensure  adequate  planning  for  life-cycle  support  of  the  program.  See  section  3.4, 
Engineering  for  life-cycle  support,  of  "Commercial  Item  Acquisition:  Considerations  and 

Lessons  Learned". 

7.10.5.4.  After  Milestone  C  or  Full-Rate  Production  Decision/Full-Deployment  Decision 
Review 

Conduct  ongoing  engineering  and  integration  for  sustainment  activities  throughout  the 
life  cycle  of  the  program. 
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